Measuring system with improved method of reading image data of an object

ABSTRACT

When a scanning start position set signal is input in an area image sensor, the content is transferred to a vertical scanning circuit, and the scan start position is set. Image of a desired row is read by horizontal scanning. Then, one shift signal for vertical scanning is input, the position of scanning is shifted by one row, and horizontal scanning is performed. Thus image of the next row is read. By repeating this operation, a desired strip-shaped image is read. The shape of the object is determined and when a portion is determined to have complicated shape, the image data is input by means of a lens having long focal length, and image data of other portions are input by means of a lens having short focal length. By putting together a plurality of input image data, image data as a whole is generated.

BACKGROUND OF THE INVENTION

[0001] 1. Field of the Invention

[0002] The present invention relates to a measuring system and, morespecifically, to a measuring system for measuring a three-dimensionalshape of an object.

[0003] 2. Description of the Related Art

[0004] Use of light-section method for measuring a three-dimensionalshape of an object has been proposed. Light-section method is based onprojection of slit shaped light on a surface of an object, andphotographing the light reflected therefrom by using an area sensor, asshown in FIG. 56 (details will be described later). A spatial coordinateof a point p of the object corresponding to one point q of thephotographed image is calculated as the coordinate of an intersectionpoint of a plane S formed by the slit shaped light and a line connectingthe point q and the center O of the taking lens. Since the spatialcoordinate of each point of the object surface irradiated by the slitshaped light can be calculated by using one slit shaped light,information of three-dimensional shape of the object as a whole can beobtained by repeating image input while scanning the object with theslit shaped light moved in a direction vertical to the longitudinaldirection of the slit.

[0005] However, in the above described apparatus, control of the slitshaped light, relation between arrangement of the area sensor and theslit shaped light, measurement output, patch up of a plurality of inputimages and so on are not sufficiently considered.

SUMMARY OF THE INVENTION

[0006] An object of the present invention is to provide a measuringsystem in which specific considerations of control of the slit shapedlight, relation between the arrangement of the area sensor and the slitshaped light, measurement output, patch up of a plurality of inputimages and so on are sufficiently made.

[0007] One of the above described object is attained by the measuringsystem of the present invention including a light projector whichprojects a slit shaped light toward an object, and an area sensor whichreceives light including the slit shaped light reflected on the object,the area sensor outputting signals from only a particular area includingthe reflected slit shaped light.

[0008] In the measuring system structured as described above, signalsare output only from a particular area, and therefore compared with asystem in which entire area of the area sensor is read, image can beread in a considerably short period of time.

[0009] The foregoing and other objects, features, aspects and advantagesof the present invention will become more apparent from the followingdetailed description of the present invention when taken in conjunctionwith the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0010]FIG. 1 is an illustration showing the principle of light-sectionmethod in accordance with the first embodiment of the present invention.

[0011]FIG. 2 is a schematic block diagram of the whole apparatus inaccordance with the first embodiment of the present invention.

[0012]FIG. 3 is a perspective view showing a schematic structure of thewhole apparatus in accordance with the first embodiment of the presentinvention.

[0013]FIG. 4 is an illustration of light intensity distributiongenerated at the plane of the object in the first embodiment of thepresent invention.

[0014]FIG. 5 is an illustration of the light intensity distributiongenerated at the light receiving plane of the photographing device inaccordance with the first embodiment of the present invention.

[0015]FIG. 6 is an illustration of the light intensity distributiongenerated at the light receiving plane of the photographing device inaccordance with the first embodiment of the present invention.

[0016]FIG. 7 is a cross section showing a structure of light emittingoptical system in accordance with the first embodiment of the presentinvention.

[0017]FIG. 8 is an illustration of the projected slit shaped light inaccordance with the first embodiment of the present invention.

[0018]FIG. 9 is a cross section showing a structure of a light receivingoptical system in accordance with the first embodiment of the presentinvention.

[0019]FIG. 10 is an illustration showing characteristics of the incidentwavelength of the color image sensor in accordance with the firstembodiment of the present invention.

[0020]FIG. 11 is an illustration showing wavelength of the receivedlight at the distance image sensor in accordance with the firstembodiment of the present invention.

[0021]FIG. 12 is an illustration showing an example of output controlfor the distance sensor in accordance with the first embodiment of thepresent invention.

[0022]FIG. 13 is an illustration of the parallax between the lightemitting system and the light receiving system in accordance with thefirst embodiment of the present invention.

[0023]FIG. 14 is an illustration of stepless control of angle ofelevation in accordance with the first embodiment of the presentinvention.

[0024]FIG. 15 is an illustration of stepwise control of angle ofelevation in accordance with the first embodiment of the presentinvention.

[0025]FIG. 16 is an illustration of minimum distance control with theangle of elevation fixed, in accordance with the first embodiment of thepresent invention.

[0026]FIG. 17 shows scope of reflected light incident on thephotographing device and the scope of scanning, in accordance with thefirst embodiment of the present invention.

[0027]FIG. 18 shows a sensor in accordance with X-Y address scanningmethod in accordance with the first embodiment of the present invention.

[0028]FIG. 19 shows a sensor in accordance with analog transfer method(at the time of interline transfer) in accordance with the firstembodiment of the present invention.

[0029]FIG. 20 shows a sensor in accordance with analog transfer method(at the time of frame transfer) in accordance with the first embodimentof the present invention.

[0030]FIG. 21 is an illustration of a sensor divided into blocks inaccordance with the first embodiment of the present invention.

[0031]FIG. 22 shows the manner of random access to the rows of theblock-divided sensor in accordance with the first embodiment of thepresent invention.

[0032]FIG. 23 is a block diagram showing a circuit structure of thewhole apparatus in accordance with the first embodiment of the presentinvention.

[0033]FIG. 24 shows a circuit for calculating position of centroid ofthe received light in accordance with the first embodiment of thepresent invention.

[0034]FIG. 25 is a flow chart showing an operation of a main routine ofthe apparatus shown in FIG. 23.

[0035]FIG. 26 is a flow chart showing the operation of a camera modeshown in FIG. 25.

[0036]FIG. 27 is a flow chart showing an operation in a shutter modeshown in FIG. 26.

[0037]FIG. 28 is a flow chart of an AF/AE subroutine shown in FIG. 27.

[0038]FIG. 29 is a flow chart showing an operation in data transfer modeshown in FIG. 26.

[0039]FIG. 30 is a flow chart showing an operation in a replay modeshown in FIG. 25.

[0040]FIG. 31 shows operation state transitions in the measuringapparatus in accordance with the first embodiment of the presentinvention.

[0041]FIG. 32 is an illustration of image patch up function inaccordance with the first embodiment of the present invention.

[0042]FIG. 33 is a flow chart showing an operation for the image patchup function in accordance with the first embodiment of the presentinvention.

[0043]FIG. 34 is an illustration showing the display for the image patchup function in accordance with the first embodiment of the presentinvention.

[0044]FIG. 35 is a flow showing the operation for the partial zoomingpatch up function in accordance with the first embodiment of the presentinvention.

[0045]FIG. 36 is an illustration showing a camera model whenphotographing is performed by using a camera universal head inaccordance with the second embodiment of the present invention.

[0046]FIG. 37 is a flow chart showing an operation for patching upthree-dimensional data photographed by using the camera universal headin accordance with the second embodiment of the present invention.

[0047]FIG. 38 is an illustration of the overlapping portions forpatching up two-dimensional images in accordance with the secondembodiment of the present invention.

[0048]FIG. 39 is an illustration of a reference window for patchingtwo-dimensional images in accordance with the second embodiment of thepresent invention.

[0049]FIG. 40 is an illustration of a search window for patching uptwo-dimensional images in accordance with the second embodiment of thepresent invention.

[0050]FIG. 41 shows a method of calculation of camera rotation angle inaccordance with the second embodiment of the present invention.

[0051]FIG. 42 is a flow chart showing an operation for continuityevaluation of patches at the junction portion in accordance with thesecond embodiment of the present invention.

[0052]FIG. 43 is an illustration of patch up of photograph data usingthe camera universal head in accordance with the second embodiment ofthe present invention.

[0053]FIG. 44 is a perspective view showing appearance of a rotary stagein accordance with the second embodiment of the present invention.

[0054]FIG. 45 is a flow chart showing an operation of patching upthree-dimensional data photographed by using the rotary stage inaccordance with the second embodiment of the present invention.

[0055]FIG. 46 is an illustration of photographing and image patch upoperations using the rotary stage in accordance with the secondembodiment of the present invention.

[0056]FIG. 47 is an illustration of patch up of the data photographed byusing the rotary stage in accordance with the second embodiment of thepresent invention.

[0057]FIG. 48 is a flow chart showing the operation of calculatingposition and attitude of the rotary stage in accordance with the secondembodiment of the present invention.

[0058]FIG. 49 is a flow chart showing an operation showing the method ofsetting the junction portion (without changing the real data) inaccordance with the second embodiment of the present invention.

[0059]FIG. 50 is a flow chart showing a method of setting the junctionportion (width change of real data) in accordance with the secondembodiment of the present invention.

[0060]FIG. 51 shows data generation points in the method of setting thejunction portion (with the change in real data) in accordance with thesecond embodiment of the present invention.

[0061]FIG. 52 is a flow chart showing an operation of patching upthree-dimensional data photographed with zooming, using the camera framein accordance with the second embodiment of the present invention.

[0062]FIG. 53 is an illustration of patch up of data photographed withzooming in accordance with the second embodiment of the presentinvention.

[0063]FIG. 54 is an illustration of re-sampling of two-dimensionalimages in accordance with the second embodiment of the presentinvention.

[0064]FIG. 55 is an illustration of re-sampling of three-dimensionalimages in accordance with the second embodiment of the presentinvention.

[0065]FIG. 56 shows a typical structure of a three-dimensional shapeinputting apparatus in accordance with a third embodiment of the presentinvention.

[0066]FIG. 57 is a block diagram showing a basic structure of theapparatus in accordance with the third embodiment of the presentinvention.

[0067]FIG. 58 shows a structure of an apparatus in accordance with thethird embodiment of the present invention.

[0068]FIG. 59 shows a structure of an apparatus in accordance with thefourth embodiment of the present invention.

[0069]FIG. 60 shows a structure of an apparatus in accordance with thefifth embodiment of the present invention.

[0070]FIG. 61 is an illustration showing the change of the photographingangle of view in the fifth embodiment of the present invention.

[0071]FIG. 62 shows an illustration showing an operation for an objecthaving depth in accordance with the fifth embodiment of the presentinvention.

[0072]FIG. 63 is an illustration showing an operation for an objecthaving depth in accordance with the fifth embodiment of the presentinvention.

[0073]FIG. 64 shows various parameters used in the fifth embodiment ofthe present invention.

[0074]FIG. 65 shows relation between scanning angle and distance to theobject plane in accordance with the fifth embodiment of the presentinvention.

[0075]FIG. 66 shows a relation between scanning speed and the distanceto the object plane in accordance with the fifth embodiment of thepresent invention.

[0076]FIG. 67 is a block diagram showing a basic structure in accordancewith the sixth embodiment of the present invention.

[0077]FIG. 68 shows a structure of an apparatus in accordance with theseventh embodiment of the present invention.

[0078]FIG. 69 shows a change in the photographing angle of view inaccordance with the seventh embodiment of the present invention.

[0079]FIG. 70 shows a object caused by the difference in received lightdistribution of the slit shaped light in accordance with the seventhembodiment of the present invention.

[0080]FIG. 71 shows a object caused by the difference in received lightdistribution of the slit shaped light in accordance with the seventhembodiment of the present invention.

[0081]FIG. 72 shows a object caused when the width of the slit shapedlight is not changed in the seventh embodiment of the present invention.

[0082]FIG. 73 shows a object caused when the width of the slit shapedlight is not changed in the seventh embodiment of the present invention.

[0083]FIG. 74 shows a structure of an apparatus in accordance with aneighth embodiment of the present invention.

[0084]FIG. 75 shows the reason why the width of the slit shaped lightchanges in the eighth embodiment of the present invention.

[0085]FIG. 76 is an illustration showing an example in which the widthof the slit shaped light changes in the eighth embodiment of the presentinvention.

[0086]FIG. 77 is an illustration showing an example in which the widthof the slit shaped light changes in the eighth embodiment of the presentinvention.

[0087]FIG. 78 is a block diagram showing an example of an exposureamount adjusting portion in accordance with the eighth embodiment of thepresent invention.

[0088]FIG. 79 is a block diagram showing another example of the exposureamount adjusting portion in accordance with the eighth embodiment of thepresent invention.

[0089]FIG. 80 is a block diagram showing another example of the exposureamount adjusting portion in accordance with the eighth embodiment of thepresent invention.

[0090]FIG. 81 is a block diagram showing another example of the exposureamount adjusting portion in accordance with the eighth embodiment of thepresent invention.

[0091]FIG. 82 is a block diagram showing another example of the exposureamount adjusting portion in accordance with the eighth embodiment of thepresent invention.

[0092]FIG. 83 shows a structure of an apparatus in accordance with theninth embodiment of the present invention.

[0093]FIG. 84 shows one slit shaped light formed by three LDs inaccordance with the ninth embodiment of the present invention.

[0094]FIG. 85 shows distribution of light intensity in the longitudinaldirection of the slit shaped light in accordance with the ninthembodiment of the present invention.

[0095]FIG. 86 shows the slit shaped light formed by one LD in accordancewith the ninth embodiment of the present invention.

[0096]FIG. 87 shows a slit shaped light formed by three LDs inaccordance with the ninth embodiment of the present invention.

[0097]FIG. 88 is a block diagram showing one example of the exposureamount adjusting portion in accordance with the ninth embodiment of thepresent invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0098] A first embodiment of the present invention will be describedwith reference to the figures. FIG. 2 is a schematical block diagram ofthe entire apparatus in accordance with the present invention. Brieflystated, the apparatus of the present invention includes a lightprojecting optical system 2 for irradiating an object 1 with laser beam,which is output from a semiconductor laser 5 and turned into a slitshaped light, and a light receiving optical system 3 for guiding theprojected laser beam to imaging sensors 24 and 12. These optical systems2 and 3 are arranged on a same rotary frame 4. In addition to theoptical systems, the apparatus includes a signal processing system forprocessing a signal output from a sensor for generating pitch-shiftedimages (details will be described later) and color images, and arecording device for recording the generated images. In FIG. 2, solidarrows denote flow of electric signals such as image signals, controlsignals and so on, while dotted arrows denote the flow of projectedlight. Details of these optical systems will be given later.

[0099] An outline of the signal processing system will be described.With respect to an image obtained by distance image sensor 12,subtraction between image 18 a when slit shaped light is projected andimage 18 b when slit shaped light is not projected is performed, andcalculation of the position of centroid of the incident light 19,calculation of pitch-shift information 20 and pitch-shift imagegenerating process 21 are performed on the image. The obtainedpitch-shifted image is utilized as an output to an output terminal 50after NTSC conversion 27, or as digital information to be transferred toan SCSI terminal 49 or an internal recording device 22. The imageobtained by a color image sensor 24 is subjected to analog processing 25and then to color image generating process 26. The resulting color imageis utilized as an output to an output terminal 51 after NTSC conversion28, or as digital information to be transferred to SCSI terminal 49 orrecording device 22.

[0100]FIG. 3 is a perspective view showing the schematic structure ofthe whole apparatus.

[0101] In this embodiment, a generating system for distance image having256 points of distance information in the lengthwise direction of theslit shaped light and 324 points of distance information in the scanningdirection of the slit will be described as an example. An LCD monitor 41provides display of a color image formed by color image sensor 24,pitch-shifted image stored in an internal or external recording device,various other information, menu for selection, and so on. A cursor key42, a select key 43 and a cancel key 44 are operating members forsetting, for example, various modes from the menu, or for selectingimages. A zoom button 45 is provided for changing focal length of thelight projecting/light receiving optical systems. An MF button 46 is formanual focusing. A shutter button 47 is for taking a distance image whenturned ON in a shutter mode, which will be described later. A drive 48such as an internal magnet-optic disc (hereinafter referred to as MO), amini disc (hereinafter referred to as MD) is provided as the storageapparatus for the picked up image. A terminal 49 is, for example, anSCSI terminal for digital input/output of signals of images and thelike. A pitched-shifted image output terminal 50 and a color imageoutput terminal 51 are provided for outputting images in the form ofanalog signals, and the images are provided as NTSC video signals, forexample.

[0102] The light projecting optical system scans the object by moving ahorizontally elongate slit shaped light in upward and downwarddirections, and the light beam from semiconductor laser 5 is directed tothe object through a rotating polygon mirror 7, a condenser lens 10, alight directing zoom lens 11 and so on. The light receiving opticalsystem picks up an image by means of a light receiving zoom lens 14, abeam splitter 15 and so on, and further by a distance image sensor 12and a color image sensor 24 arranged on a light receiving image pickupplane. Details of the optical systems and the imaging system will begiven later.

[0103] The slit shaped light from the light projecting system is moveddownward one pixel pitch by one pixel pitch of the distance image sensor12, by means of constantly rotating polygon mirror 7, while distanceimage sensor 12 accumulates one image. The distance image sensor scansthe accumulated image information, provides an output, and performsaccumulation of the next image. From the image provided at one input,distance information of 256 points in the lengthwise direction of theslit shaped light can be calculated. Further, by repeating the mirrorscanning and taking of images 324 times, a distance image consisting of256×324 points is generated.

[0104] As for the distance range to the object measured by one slitshaped light, the minimum and maximum measurement distances are limited,and therefore the range of incident light which is the slit shaped lightreflected by the object and entering the image pickup device is limitedwithin a certain range. This is because the light projecting system andthe light receiving system are arranged apart from each other by a baselength (length:1). This is illustrated in FIG. 17 in which Z axisrepresents a direction verticle to the image pickup plane for thedistance image. The position of the dotted line d is a reference planefor measurement, and the distance from the plane of the devicecorresponds to d.

[0105] Therefore, in the measuring apparatus, the position of thecentroid of the laser beam received at 256 lines is calculated based onthe input image. More specifically, the position of the centroid iscalculated as the amount of deviation from the reference plane formeasurement, that is determined based on an object distance output froman auto focus unit and direction of the projected slit shaped light,that is, the time from the start of scanning. The calculation of theamount of pitch shift will be described with reference to FIG. 4. FIG. 4shows light intensity distribution generated by the slit shaped lightdirected to the object. The sections at the lower portion of the figurerepresent areas monitored by each of the elements of the distance imagesensor. These sections have numbers 1, 2, 3, 4, . . . allotted thereto,starting from the front side. A slit shaped light having very narrowslit width is moved for scanning only by 1 pitch of the distance imagesensor by the rotation of polygon mirror 7 while one image isaccumulated. Therefore, the light intensity distribution when one imageis input corresponds to a rectangular light intensity distribution ofwhich width corresponds to 1 pitch of the distance image sensor.

[0106] In order to calculate distance information in the direction ofthe Z axis for each pixel of the distance image sensor, such arectangular light intensity distribution having the width of 1 pitch isdesirable. When the width of the light intensity distribution becomeswider than 1 pitch, the distance information measured would becalculated as weighted mean of the intensity of light received atadjacent areas, and hence correct distance information would not beobtained.

[0107] Assume that there is a step-shaped object surface such asrepresented by the dot in FIG. 4, and a slit shaped light is directedfrom a direction vertical to the plane of the object. The thinrectangular parallelepiped represents the light intensity distributionof the slit shaped light and the hatched area represents the slit-shapedimage irradiated by the light beam. When we assume a positional relationin which an optical axis Oxp of the light receiving optical system isprovided inclined to the left from an optical axis Oxa of the lightprojecting system, the light intensity distribution of the received slitshaped light at the light receiving plane would be as shown in FIG. 5,because of a filter, which will be described later. It is desirable toremove fixed light component other than the laser beam component so thatthe fixed light component is not included in the receive lightintensity. For this purpose, an image irradiated with the laser beam andan image not irradiated with the laser beam are both input, and thedifference therebetween is used. The sections at the lower portionrepresent respective element regions of the distance image sensor. Infront of the distance image sensor, there is provided an anisotropicoptical filter which does not degrade resolution in the lengthwisedirection of the received slit shaped light but degrades the resolutionin the widthwise direction of the slit shaped light, and by means ofthis filter, the light intensity having such a Gaussian distribution asshown in FIG. 5 results. With respect to this light intensitydistribution, by calculating the centroid of the light intensitydistribution from respective sensors for columns 1, 2, 3, 4, . . . , theposition at which the light is received can be calculated with higherresolution than the pixel pitch. The reason why the width of the slitshaped light incident on the sensor is not narrowed but selected to havethe width of about 5 to 6 pixels by using a filter for detecting theposition at which the slit shaped light is received is that when thewidth of the incident slit shaped light becomes narrower than the widthof one pixel, the resolution for detecting the position could be at mostthe same as the pixel pitch.

[0108] Based on the light intensity distribution D1 received by thefirst column, the position G1 of the centroid of the first column iscalculated. In the similar manner,, the positions G2, G3, G4, . . . ofcentroid of the second, third, fourth and the following columns arecalculated, and thus the centroid of each column is calculated. As shownin the figure, the optical axis of the light projecting system isvertical to the plane of the object. However, the optical axis of thelight receiving system is inclined to the left. Therefore, when theobject has a step as shown in FIG. 4, the centroid of the higher portion(third and fourth columns) is positioned shifted to the right, withrespect to the centroid of the lower portion (first and second columns).Though the distribution D1 of the first column and distribution D4 ofthe fourth column only are shown in FIG. 5, the distribution D2 of thesecond column is the same as the distribution D1 of the first column,and the distribution D3 of the third column is the same as thedistribution D4 of the fourth column. The relation between the lightintensity distribution and the positions of the centroid is representedtwo dimensionally in FIG. 6. Since the distributions of the first andsecond columns are the same, the calculated center of gravities G1 andG2 are detected as the same position and since the distributions of thethird and fourth columns are the same, the calculated center ofgravities G3 and G4 are detected as the same position.

[0109] In this manner, from a slit-shaped image corresponding to oneslit, positions of the incident light at 256 points are calculated. Byperforming similar calculation for the slits directed to 324 directions,324 images are obtained, and a pitch-shifted image consisting of 256×324points is obtained. The obtained pitch-shifted image consists of onlythe positional information of the slit shaped light. Therefore, in orderto obtain an accurate distance image, calibration (correction) based ona table of detailed data such as lens aberration correction isnecessary. Therefore, lens aberration estimated from the focal length fand in-focus position d of the taking lens is calculated, corrected, anddistortion in the longitudinal and lateral directions with respect tothe camera is corrected. Similar operation is performed with respect tothe color image. The data necessary at that time includes information ofvarious measurement lenses, that is, focal length f and in-focusposition d. In the system of the present embodiment, calibration isperformed on a computer system, and connection to the measurementapparatus of the present invention (shown in FIG. 3) is provided by SCSIterminal, for example. Alternatively, the data may be shared by using arecording medium such as MO.

[0110] In this manner, from the body of the measuring apparatus, colorimages and pitch-shifted images are provided as digital signals from aterminal such as SCSI terminal, or provided as analog video signals froman output terminal such as BTSC terminal. Data necessary for calibrationare provided to the computer as digital signals from SCSI, for example.When a drive 48 such as internal MO or MD is to be used, images andvarious data are recorded on the recording medium. The takenpitch-shifted images and color images are transferred to a computerconnected to the measuring apparatus, together with various taking lensinformation. In the computer, based on the transferred pitch-shiftedimages and the taking lens information, the data are calibrated andconverted to a distance image having information with respect to thedistance to the object. As for the pitch-shifted image, aftercalibration, a conversion curve with respect to the stored amount ofshifting and measured distance is extracted for every XY position,longitudinal and lateral positions on the image plane, focal length fand in-focus position d, and based on the conversion curve, thepitch-shifted image is converted to a distance image.

[0111] Conversion to the distance image is well known and the detailedare described, for example, in Institute of Electronics, Information andCommunication Engineers, Workshop Material PRU 91-113, Onodera et al.,“Geometrical Correction of Image Without Necessitating CameraPositioning”, Journal of Institute Electronics, Information andCommunication Engineers, D-II vol. J74-D-II, No. 0, pp. 1227-1235, '91/9Ueshiba et al, “Highly Precise Calibration of a Range Finder Based onThree-Dimensional Model of Optical System.”

[0112] The measuring apparatus in accordance with the present inventionwill be described in greater detail.

[0113] First, the optical system will be described. Referring to FIGS. 1and 2, when a distance image is photographed, a slit shaped light S isdirected to an object 1, from a slit shaped light projecting apparatus(light projecting optical system) 2. Slit shaped light projectingapparatus 2 includes a light source, for example a semiconductor laser5, a collective lens 6, a polygon mirror 7, a cylindrical lens 8, acondenser lens 10 and a light projecting zoom lens 11. In stead of apolygon mirror 7, a rotary mirror such as a resonance mirror, galvanomirror or the like may be used.

[0114] Cylindrical lens 8 has not spherical but columnar convex surface.Therefore, it does not provide a point of focus but a line of focuswhich is parallel to the axis of the column. Polygon mirror 7 has anumber of mirrors provided around an axis of rotation, and by rotation,light beams incident on respective mirror surfaces are moved in onedirection successively for scanning.

[0115] The structure of the light projecting optical system will bedescribed with reference to FIG. 7. FIG. 7(a) is a side view of thelight projecting system, and FIG. 7(b) is a top view thereof. In FIG.7(b), such portions that overlap when plotted two dimensionally arepartially omitted. Referring to FIG. 7(a), the slit shaped light has itslength in a direction vertical to the sheet. The solid line fromsemiconductor laser 5 to condenser lens 10 represents the optical path.After condenser lens 10, dotted lines are phantom lines representingposition of re-imaging of the slit shaped light. In FIG. 7(b), the slitshaped light has its length in the upper and lower directions of thefigure. The solid line from semiconductor laser 5 to cylindrical lens 8represents the optical path. Dotted lines after condenser lens 10 arephantom lines indicating the position of re-imaging. Chain dotted linebetween cylindrical lens 8 and condenser lens 10 schematically shows themanner how the laser beam which has progressed as points is converted toa slit shaped light having a certain width by means of cylindrical lens8. The slit shaped light is re-imaged at a position represented by aline (two-dotted chain) at the left end of FIGS. 7(a) and 7(b).

[0116] Collimator lens 6 (corresponding to collective lens 6 of FIG. 2)has a lens power sufficient to collect light beam (having the emissionwavelength of 670 nm, for example) output from semiconductor laser 5onto the condenser lens. The laser beam which has passed throughcollimator lens 6 is reflected to a direction vertical to the lens ofthe slit shaped light, by means of polygon mirror 7. This deflectionenables scanning of the object plane with the slit shaped light. Thelaser beam deflected by polygon mirror 7 first enters the fθ lens 29.The fθ lens 29 is arranged for correcting non-linear component, sincethe speed of movement of the slit shaped light on the object surface isnon-linear with respect to the constant speed of rotation of polygonmirror 7.

[0117] The subsequent collimator lens 30 directs the luminous fluxentering condenser lens 10 from the direction of scanning by the polygonmirror 7 to a direction vertical to the condenser lens, so as to improveefficiency of projection. The laser beam is converted to a slit shapedlight having its length extending in the horizontal direction (verticalto the sheet of FIG. 7(a)) by means of cylindrical lens 8, collectedonto a pupil plane of condenser 10 and forms an image there, so that itis directed to the object as a very narrow slit shaped light.

[0118] The slit shaped light once imaged by condenser lens 10 arrangedon image plane (imaging surface) 10 p of light projecting zoom lens 10passes through light projecting zoom lens 11 and directed to the object.The size of the image plane is selected to match the size of the imagepickup device, for example ½ inch, ⅓ inch or the like. In theembodiment, it is selected to be ½ inch. The slit shaped light has itslength in the horizontal direction, generated by cylindrical lens 8, andit is moved for scanning at high speed in accordance with the rotationof the polygon mirror, in a direction vertical to the length of thedirected slit shaped light. At this time, the in-focus position of thelight projecting zoom lens is controlled by an AF driving system 17based on a signal from an auto focus sensor 31 provided in thephotographing system, simultaneously with and to have the same value asthe photographing system in accordance with the distance to the plane ofthe object. Auto focus sensor 37 is one commonly used for a stillcamera. The focal length is also controlled based on the operation by auser or from the system, simultaneously with and to have the same valueas the photographing system.

[0119] Polygon mirror 7 is connected to projecting scanning drivingsystem 9 including a polygon mirror driving motor and a polygon mirrordriver, and its rotation is controlled by this system. A scanning startsensor 33 is a sensor employing a photodiode arranged aside condenserlens 10, and it monitors whether laser scanning has reached a stablestate, that is, the timing for starting scanning.

[0120] The light projecting system has a zooming function which allowsadjustment of necessary magnification with respect to the object 1. Thezooming function includes power zooming (PZ) in which user canarbitrarily select the angle of view, and automatic zooming (AZ) inwhich pre-selected field of view is automatically attained. With respectto the zooming of light receiving optical system 3, light projectingoptical system 2 is controlled by an AZ driving system 16 so that theangle of view is constantly matching, and zooming is performed so as toprovide equal optical magnification constantly. The relation betweenzooming and projection of slit shaped light is represented by theequations (1) to (3) below, with reference to the schematic diagram ofFIG. 8.

θ=α1×1/f  (1)

φ=α2×1/f  (2)

ψ=α3×1/f  (3)

[0121] Using a point in the light projecting system as a reference, θrepresents an angle of movement of the very narrow slit shaped lightwhile one image is integrated, in order to obtain a column of 256 pointsof pitch-shifted image; φ represents an angle indicating length of theslit shaped light on the object; and ψ represents total scanning angleof 324 times of the slit shaped light on the object. The slit shapedlight scans, starting from the position denoted by the solid line to thedirection of the arrow until it reaches the position denoted by thedotted line. The reference character f represents focal length of thelight projecting lens. The width of the slit shaped light itself is setas narrow as possible. Reference characters α1, α2 and α3 representproportional coefficients and these angles θ, φ and ψ, are proportionalto the reciprocal number of focal length f.

[0122] In a vertical direction of slit shaped light projecting apparatus(light projecting optical system) 2, and apart from slit shaped lightprojecting apparatus 2 by base length 1, a photographing apparatus(light receiving optical system) is provided arranged on one rotaryframe 4, which apparatus includes a color image photographing system anda distance image photographing system. The structure of the lightreceiving optical system is shown in FIG. 9. Light receiving opticalsystem 3 includes a photographing zoom lens 14, an auto focusing unit31, a beam splitter 15, various filters 61 and 62, a color CCD imagepickup device 24, and a distance image sensor 12. The received light issplitted by beam splitter 14 s arranged in photographing zoom lens 14,and one part of the light is directed to auto focus unit 31. The AF unit31 measures approximate distance to the object plane and adjusts thepoint of focus of the light projecting system and light receiving systemlenses. In this embodiment, a common unit used in a video camera, asingle lens reflex camera or the like is used.

[0123] The other luminous flux splitted by beam splitter 14 s arrangedin photographing zoom lens 14 is further splitted into transmitting andreflecting two luminous fluxes by a beam splitter 15 arranged behind thephotographing zoom lens, and guided to distance image sensor 12 andcolor image sensor 24, respectively. Beam splitter 15 has such anoptical characteristic that transmits long wavelength component ofluminous flux entering the distance image sensor 12, in this embodimentwavelength component longer than about 650 nm including laser wavelengthcomponent (679 nm), and that reflects other wavelength components.

[0124] The reflected short wavelength component includes most of thewavelength components of visible light. Therefore, generally, it doesnot affect color information. The reflected luminous flux passes througha lowpass filter 61 such as a crystal filter for preventing spriousresolution, and imaged on a single plate color image sensor 24. Thesingle plate color image sensor 24 is a CCD image pickup device used ina video camera or the like, on which RGB or yellow Ye, cyan Cy, magentaMg and green G of complementary color system are arranged as a mosaic,for extracting color information. Green may be used as a luminancesignal. FIG. 10 shows wavelength band of the light received by colorimage sensor of the complementary color system. The color image sensorreceives the light in the wavelength range reflected by a beam splitterhaving the reflectance of h. The curves are spectral sensitivity of thepixels with color filters of yellow Ye, cyan Cy, magenta Mg and green G.

[0125] The luminous flux having long wavelength component transmittedthrough beam splitter 15 further passes through a filter for cuttinginfrared ray (hereinafter referred to as IR) for extracting only thelaser beam component (having the wavelength of 670 nm), and furtherpasses through a lowpass filter such as a crystal filter, and is imagedon distance image sensor 12. In FIG. 9 showing the structure of lightreceiving optical system, the IR cutting filter and the lowpass filterare represented by one filter 62. FIG. 11 shows the wavelength band(hatched portion) of the light beam received by distance image sensor12. The shorter wavelength region than the laser beam wavelength is cutby the beam splitter 15 (having such transmission factor as representedby the solid line) and the longer wavelength region is cut by IR cutfilter 62 (having such transmission factor as represented by the dottedline).

[0126] Lowpass filter 62 used here is not for preventing spriousresolution of color images mentioned above, but for providinginterpolating function for detecting positions of the received laserbeam with a resolution finer than the pitch of the imaging devices, forcalculating the distance data. For this purpose, it should preferablyhave anisotropic optical characteristic which does not degraderesolution in the lengthwise direction of the received slit shaped lightbut degrades the resolution in the widthwise direction of the slitshaped light, different from the isotropic optical characteristic of thelowpass filter 61 for the color image. As means for realizing suchoptical characteristic, a single layer crystal filter or a lowpassfilter utilizing diffraction such as grating may be used. However, thelowpass filter is not essential in the system structure and the functioncan-be provided by an analog filter for subsequent sensor output, or bya digital filter after digital conversion of the sensor output.

[0127] Scanning of image pickup devices 12 and 24 will be described. 12p and 24 b shown adjacent to image pickup devices 12 and 24 of FIG. 9are plan views of the image pickup devices 12 and 24 for easierunderstanding. Generally, speed of scanning the CCD image pickup devicein the vertical direction is lower than the scanning in the lengthwisedirection (horizontal direction) along horizontal registers 12 h and 24h. Therefore, color image obtained by image pickup device 24 (24 p) issubjected to analog signal processing in accordance with an output fromhorizontal transfer line of the CCD scanning at high speed, andconverted to NTSC signal successively so that image output can beprovided to the monitor. When the pitch-shifted image is to be output tothe same monitor, it is preferable to generate distance image data inthe same direction and same positional order as the horizontal scanningdirection of the color image, since in that case it becomes unnecessaryto store positional information, and hence the necessary memory capacitycan be reduced and function required of the memory can be simplified.

[0128] Accordingly, it is preferred that the direction of the length ofthe slit projected for measuring distance data is the same as thedirection of high speed scanning of the image sensor for color images.In other words, generally it should be in the horizontal scanningdirection. Further, the scanning direction of the slit should bevertical scanning direction of the color image. Namely, the projectedslit shaped light should preferably be moved upward and downward forscanning.

[0129] Therefore, in such a three-dimensional shape measuring apparatusor a three-dimensional input camera as that of the present inventionutilizing light section method in which the object captured by twodifferent image input sensors, that is, color image sensor and distantimage sensor, a slit-shaped beam of which length extends in thehorizontal scanning direction of the color image pickup device should beprojected and slit should be moved in the same direction as the verticalscanning direction of the color image pickup device for scanning,whereby the memory can be reduced and the requirements for the memorycan be released. Further, in order to read slit-shaped images at highspeed from the distance image sensor with respect to such a slit-shapedbeam, the direction is limited. Therefore, the horizontal directionallowing high speed scanning by the distance image sensor must beparallel to the horizontal scanning direction of the color image sensor.Therefore, the relation between positions of these image sensors and theincident slit shaped light and the relation between the scanningdirections are as shown in FIG. 9.

[0130] It is effective for the optical system having such photographingsystem structured as described above to equip the following twostructures.

[0131] Namely, the color image and the image for generating distanceimage are input through the same lens. However, the light intensityobtained from the wavelength for the color image is not related to thelight intensity obtained from the wavelength of the distance image.Therefore, exposure light intensity is desirable to independentlycontrolled. When a close object is to be measured in the dark,brightness for distance is high while brightness for color image is low.When an object at a distance is to be measured with sufficientillumination, the brightness for the distance is low, while brightnessfor the color image is high. Therefore, in the light receiving zoomlens, control of the exposure is not effected by the diaphragm which isa common exposure adjusting means for general lens, and the diaphragm isfixed at the open state.

[0132] Exposure control of the color image is effected by an electronicshutter function of a generally used FIT-CCD or the like, in whichexposure is adjusted in accordance with the time of accumulation.Generally, electronic shutter function of the FIT-CCD or the like usedas the color image sensor allows accumulation time control of {fraction(1/60)} to {fraction (1/10000)} sec. In order to ensure wider dynamicrange, an ND filter for reducing the amount of transmitted light whilenot changing components of the incident light may be insertedimmediately before the color image sensor when it is used outdoor withsufficient light. By doing so, the amount of light incident on thesensor can be reduced so that it can be used at higher brightnesswithout decreasing the amount of light entering the distance imagesensor.

[0133] As for the exposure control for the distance image, laserintensity is adjusted by controlling the number of used lasers forprojecting light, controlling current supply to the laser, andcontrolling insertion of the ND filter at an arbitrary optical positionfrom the laser to the output lens, or the output level is adjusted bythe amplifier gain supplied to the output signal. In this control, thevalue for controlling laser intensity is determined based on thedistance information Daf to the object obtained from the AF controlportion, and focal length f of the lens under the measuring conditions.FIG. 12 shows an example of the control map.

[0134] Generally, the output of the distance image sensor is in reverseproportion to the square of distance information Daf to the object. Whenthe focal length f becomes shorter, the area which needs illuminationbecomes larger, and therefore the output signal of the distance imagesensor becomes smaller. Therefore, in the apparatus of the presentembodiment, the output level of the data for calculating distance imageis controlled with the number of lasers changed in accordance with thefocal length. In the example of FIG. 12, three lasers are used for thefocal length f of up to 36.7 mm, and one laser is used for longer focallength. It is further controlled by changing amplifier gain provided byan analog pre-processing circuit to the output of the distance imagesensor, in accordance with image magnification β(=daf/f) calculatedbased on the focal length f and the distance information Daf to theobject determined by the output from the AF sensor. In the exampleshown, the amplifier gain is set to be ½ when β-35 to 50, 1 when β=50 to75, 2 when β=75 to 100 and 4 when β-100 to 200. Further, when higherlaser beam is used for measuring in a telephoto region having long focallength for a close object, the laser intensity can be effectivelycontrolled by inserting an ND filter at an arbitrary optical positionfrom the laser to the output lens.

[0135] However, when satisfactory result of measurement cannot beobtained by using the values controlled in the above described manner,it is possible to provide a laser intensity adjusting key for adjustingthe laser intensity by key operation, or to change sensor accumulationtime. Alternatively, laser prescanning may be performed based on anestimated laser intensity control value obtained based on the distanceinformation and the estimated reflective index of the object. Morespecifically, the maximum output value of the distance image sensor atthe time of prescanning is calculated. The laser intensity and imagesensor accumulating time which are within the dynamic range of the A/Dconversion and sufficient for calculating distance information in thesucceeding stage are calculated. Thus the distance image is taken basedon the calculated control values. If an auxiliary illumination isavailable for auto focusing, it is possible to detect by the AF sensor,the amount of reflected light derived from the auxiliary illuminationwith respect to the center of the field of view at which the object isconsidered to be existing, and to calculate laser intensity and imagesensor accumulation time based on the detected reflected amount of lightfor taking the distance image.

[0136] Additionally, there is inevitably generated a parallax becauselight is projected and received at different points of view (positions)(see FIG. 13). Therefore, it is effective to equip means for solvingthis parallax. When light is projected and received by the same lenssystem having same image plane size and the same focal length, the fieldof view matches only at a specific distance (as denoted by a largerarrow OBJl). When there is not an object at the distance where the fieldof view matches, three-dimensional shape of a region to which light isnot projected would be measured, and therefore measurement becomesimpossible. For example, when an object at a position where the fieldsof view do not match as represented by a small arrow OBJ2 of FIG. 13 isto be measured, the scope of light projection is different from thescope of light reception, and therefore the light receiving system scansthe region denoted by the upper end of the arrow, to which light is notprojected.

[0137] The above-described problem can be solved by the followingstructure.

[0138] (1) The angle of elevation of the optical axis of the lightprojecting system is changed in stepless manner in accordance with thedistance to the object (see FIG. 14). The light projecting system andthe light receiving system are set to have the same focal length. Theangle of elevation of the optical axis (denoted by the dotted line) ofthe light projecting system is changed in accordance with the distanceto the object based on auto focus measurement, so as to meet the scopeof scanning of the light receiving system, which is fixed. Morespecifically, since the influence of parallax becomes serious as thedistance is smaller, the angle of elevation is enlarged to set thescanning scope at S1, and the angle of elevation is made smaller forgreater distance and the scanning range is set at S2. The optical axisof the light projecting system is changed mechanically.

[0139] (2) The angle of elevation of the optical axis of the lightprojecting system is changed continuously in accordance with thedistance to the object, by some optical means such as a prism havingvariable refractive index immediately after emission of light from thelight projecting lens unit. Here, the light projecting and lightreceiving systems are set to have the same focal length f. By insertingand ejecting a prism having a curvature in accordance with the distancebased on auto focus measurement, the refractive index is changed andhence the angle of elevation of the optical axis of the light projectingsystem is changed.

[0140] (3) The focal length fa of the light projecting system iscontrolled so that it becomes smaller than the focal length fp of thelight receiving system, by using an optical system having the same imageplane size. Alternatively, an optical system having larger image planesize is used for the light projecting system so that the lightprojecting and receiving systems have the same focal length fa and fp.By using such means, there is provided a margin for the scanning scopeof the light transmitting system with respect to the scanning scope ofthe light receiving system (about 1.5 times that of the light receivingsystem), and at the same time, the distance to the object is dividedinto a plurality of zones, and the angle of elevation of the opticalaxis of the light projecting system is changed stepwise, correspondingto respective zones. In the example shown in FIG. 15, the distance tothe object is divided into two zones, and the farther zone is denoted byzone Z1 and the closer zone is denoted by Z2. For the farther zone Z1,the angle of elevation of the optical axis of the light projectingsystem is changed by a prescribed angle, and for the closer zone Z2, theangle of elevation of is changed by a larger angle than in zone Z1.

[0141] (4) Similar to the option (3) above, there is provided a marginin the scanning scope of the light projecting system (of about 1.5 timesthat of the light receiving system), the angle of elevation of theoptical axis of the light projecting system is fixed, and there isprovided a limit in the closest measurable distance, in accordance withthe focal length. In the example shown in FIG. 16, at a position nearerthan the position of the arrow OBJ2, the light receiving area does notcoincide the light projecting area, and therefore the distancecorresponding to this position denoted by the arrow is set as theclosest distance.

[0142] In options (1) and (2) above (FIG. 14), it is assumed that thefields of view completely coincide with each other. Therefore, it ispossible to drive the distance image sensor simultaneously with thestart of laser scanning and to start taking the image. Meanwhile, in theoptions (3) and (4), the fields of view are not coincident as shown inFIGS. 15 and 16 and the laser scanning area by the light projectingsystem is wide, resulting in unnecessary region. Therefore, the timerequired for scanning this unnecessary region is calculated based on theauto focus calculation reference distance. Since scanning precedes froman upper side to the lower side, there is an unnecessary region at thestart of scanning. Therefore, microcomputer is set to start taking ofdata from distance image sensor after the lapse of the aforementionedcalculated time. In that case, since the scanning range is wider, thetime necessary for laser scanning is about 1.5 times that of the options(1) and (2), and therefore time for the input of the three-dimensionalshape becomes longer by that time.

[0143] Since the angles of elevation of the light projecting system andthe light receiving system differ from each other, laser is moved notstrictly at the same speed on the surface of the object which isvertical to the optical axis of the light receiving system. Morespecifically, the laser scanning is dense at the lower side of theobject and sparse at the upper side of the object. However, since theangle of elevation itself is very small, it does not present seriousproblem. By providing a conversion table from positional information inthe vertical direction scanned by the sensor and the amount of pitchshift to the distance information, an approximately isotopicalthree-dimensional measurement is possible.

[0144] The sensingsystem will be described in greater detail.

[0145] When there is a limit in the distance range to the object to bemeasured with respect to the direction of one projected slit shapedlight, the position on the sensor receiving the light reflected by theobject is also limited within a certain range. This is illustrated inFIG. 17.

[0146] In the figure, Df represents maximum distance for measurement andDn represents minimum distance for measurement. Now, if the plane cut bythe slit shaped light projected from the light projecting system is slitA, the scope on the plane of the image pickup device receiving the slitshaped light reflected by the surface of the object is limited to aclosed area Ar, in which a position of projection on the image pickupdevice of the three-dimensional position of an intersection PAn betweenthe minimum distance Dn for measurement and the slit A is the lowermostpoint in the figure, and the projected point on the image pickup deviceof the three-dimensional position of the intersection Baf between themaximum distance Df for measurement and slit A, projected on the imagepickup device with the position of the main point of the image pickupsystem being the center, is the uppermost point in the figure. Assumingthat the light projecting system and the light receiving system have thesame positional relation, in case of slit B, the scope on the plane ofthe image pickup device is limited to a closed area Br on the imagepickup device, in which the point of projection of the intersection PBnof the minimum distance for measurement Dn and slit B is the lowermostposition in the figure, and the point of projection of intersection PDfof the maximum distance for measurement Df and the slit B is theuppermost point in the figure.

[0147] In this manner, in order to generate a column for distance dataconsisting of 256 points by projecting one slit shaped light, not theentire area of the image pickup devices but only the necessary areacorresponding to the slit shaped light is scanned, and therefore thespeed of processing can be increased.

[0148] In order to increase the speed of operation of the apparatus forgenerating data of a three-dimensional shape, a function of outputtingat high speed a strip shaped image of the corresponding area only, forexample only the image of 256×16 pixels is desired. An high speed drivensolid state image pickup device allowing selective reading of such stripshaped region includes the following three types of solid state imagepickup devices. The first option is addition of a read start addresssetting function to an image pickup device having X-Y address scanningsystem such as a MOS and CMD (FIG. 18). The second option is addition ofa function of discharging in parallel with charge transfer to a read-outtransfer path (generally, a horizontal register), in an analog transfersystem such as a CCD image pickup device (FIGS. 19, 20). The thirdoption is setting beforehand blocks divided into strips regardless ofthe scanning method, providing an output function for each block, andutilizing parallel outputs thereof (FIG. 21).

[0149] A structure of a sensor employing the X-Y address scanning methodas the first option is shown in FIG. 18. Generally, scanning of pixelsis performed by switches arranged in a matrix of a vertical scanningcircuit 61 and a horizontal scanning circuit 62. The vertical andhorizontal scanning circuits 61 and 62 are formed of digital shiftregisters. By inputting 256 horizontal shift signals for one shiftsignal input of vertical scanning, one row (256 pixels) can be scanned.In this embodiment, by providing a scan start set register 63 forsupplying a scan start set signal, that is the register initial value,to vertical scanning circuit 61, strip-shaped random access reading isrealized. To the scan start set register 63, signals sgn1 and sgn2indicative of the scanning start position are input, as an instructionof the position at which strip shaped image is to be read out.

[0150] Now, if the number of pixels is increased, the number of bits ofthe scan start set signal is also increased, resulting in larger numberof input pins. Therefore, it is preferable to provide a decoder 64 forthe scan start set signal. By parallel transfer of the content in scanstart set register 63 to vertical scanning circuit 61 at the start ofreading, the position for starting scanning (row) is set. By repeating256 horizontal scanning, signals from the desired row can be obtained.Then, 1 shift signal input for the vertical scanning and 256 shiftsignal inputs for the horizontal direction are performed. to read thesignals of the next row. By repeating this operation, the image of thedesired strip shaped region is read. By the above described operation,scanning of the desired strip shaped area only can be realized. Thusnecessary scanning can be completed in far shorter time period (numberof rows read out/number of rows of the entire area) than the timenecessary for scanning the entire region.

[0151] The region which is once readout is reset and the nextaccumulation is started. However, in a region which has not yet beenread out, charges are continuously accumulated. At this time, the nextreading is from the same region, there is no problem. However, when thenext reading is from a different region, there would be imageinformation having different accumulation times. In three-dimensionalmeasuring apparatus using light-section, it is necessary to read whileshifting the strip-shaped region which needs reading, together with thescanning of the laser slit. In a region which is read out repeatedly,the image corresponding to the time of integration from the last readingto the present reading is read out. However, as the read region isshifted, in the region which is newly read out, an image would beprovided which corresponds to thoroughly continued integration.Therefore, in the present invention, the strip-shaped region for readingis set such that it includes both the region necessary at this time andthe region necessary for the next time. By doing so, the region which isnecessary for the next input has its integration cleared without fail atthe last reading. Therefore, taking of an image consisting of pixelshaving different integration times can be avoided.

[0152] A structure for interline transfer of the CCD image pickup deviceand a structure for frame transfer are shown in FIGS. 19 and 20,respectively, as the second option. In the CCD image pickup device inaccordance with the present embodiment, an integration clear gate ICGfor discharging the charges to an overflow drain OD is provided parallelto a transfer gate TG for parallel charge transfer to a horizontalregister 63, thus realizing strip-shaped random access reading.

[0153] In case of interline transfer, generally, the charges accumulatedin every pixel are transferred in parallel from the light receivingportion to the transfer region, at the time of completion of imageaccumulation for the entire area. As for the scanning of the chargesgenerated in each of the pixels, one shift signal is input to thevertical register and the transfer gate TG, charges in the verticalregister are shifted downward one stage by one stage, and the charges inthe lowermost vertical register are read to the horizontal register 66.Thereafter, by supplying 256 shift signal inputs of the horizontal shiftsignal, charges of one row can be scanned. By repeating this operationfor the number of rows (340 rows), reading of the entire region isperformed.

[0154] In the present embodiment, charges generated at an unnecessaryrow in the step of scanning charges generated at respective pixels aredischarged to the overflow drain OD in parallel, by supplying a 1 shiftsignal input to the vertical register and to the integration clear gateIC1. For the row which needs reading, a 1 shift signal input is providedfor the vertical register and the transfer gate TG so as to shiftcharges of the vertical register downward one stage by one stage inparallel and the charges in the lowermost vertical register is read tothe horizontal register 66. Thereafter, by supplying 256 shift signalinputs of the horizontal shift signal, charges of one row are scanned.In this manner, random access function on row by row basis is realized,and necessary scanning can be completed in far shorter time period thanthe time necessary for scanning entire region by the image pickup device(number of rows to be read out/number of rows of the entire region).

[0155] In the case of frame transfer shown in FIG. 20, the structure islarger than that of interline transfer. The upper portion is aphotoelectric conversion region and the lower side is accumulationregion. Generally, the accumulation region has the same number of pixelsas the photoelectric converting portion. In normal operation, theaccumulated charges of all pixels are transferred in parallel from thephotoelectric converting region to the accumulating region by verticaltransfer pulses of which number corresponds to the number of rows, atthe time when image accumulation of the entire region is completed.After the transfer, the scanning of charges generated at respectivepixel is performed in the same manner as in the interline transfer. Morespecifically, charges are read to the horizontal register 66 by thecontrol of the vertical register and the transfer gate TG, andthereafter 256 shift signals for horizontal shifting are input, so thatcharges of one row can be scanned.

[0156] In this embodiment, the accumulated charges of the pixels of theentire region are transferred in parallel by vertical transfer pulsesthe number of which corresponds to the number of rows, from thephotoelectric converting region to the accumulating region. Thereafter,at the time of transfer to the horizontal register 66, charges ofunnecessary rows are discharged to the overflow drain OD in parallel,simply by inputting one shift signal to the vertical register and theintegration clear gate ICG, in the step of scanning charges generated atrespective pixels. Meanwhile, as for the accumulation region, only thenecessary number of rows (for example 16 rows) may be prepared for everyreading, and as for the signal for the unnecessary rows of pixels of thefirst reading, it may be synchronized with the vertical transfer pulsefor vertical transfer from the photoelectric converting region to theaccumulating region so as to open the integration clear gate ICG todischarge the charges, and only the charges of the rows of pixels whichneed reading are transferred to the accumulating region and read fromthe horizontal register 66. By doing so, random access function on rowby row basis is realized and necessary scanning is completed in farshorter time period than the time necessary for scanning the entireregion (the number of rows to be read out/the number of rows of theentire region).

[0157] A structure of a sensor in which a plurality of blocks areprepared by division and the output is given block by block is shown inFIG. 21 as the third option. Here a sensor using X-Y address scanningmethod will be described as an example. However, the same structure canbe also employed in an analog transfer method such as in the CCD imagepickup device. In the present embodiment, a number of blocks of whichnumber corresponds to the preset number of rows necessary for readingare prepared, and the signals of respective blocks are scanned inparallel and output. With respect to the parallel readout output, outputis selected by operating a multiplier 65 in accordance with the regionto be read out, and thus final output is obtained. By such reading,random access on row by row basis is realized, though the order ofoutputs is different. The time for reading can be compressed by thenumber of block division. The relation between the manner of output ofthe strip-shaped image read at random by the block-divided structure andthe signals for switching blocks of the multiplier is shown in FIG. 22.In the figure, the reference numerals 1 to 16 correspond to line numbersof FIG. 21.

[0158]FIG. 21 shows a very simple example in which there are two blocks(B1 and B2) and arbitrary three rows are read. Description will be givenwith reference to FIG. 21 and FIG. 22 showing the relation of the outputsignals. The sensor includes two different outputs therein, namely, ablock B1 output (FIG. 22.a) providing lines 1 to 3, and a block B2output (FIG. 22.b) outputting lines 4 to 6. These are transmitted asanalog signals to the multiplier, selected in accordance with aselection signal Se1 and output. By the operation of multiplier 65, whenblock B1 output is selected as the sensor output Out, the output fromblock B1 is used as the sensor output as it is, and outputs ofstrip-shaped images of lines 1, 2 and 3 are output successively (FIG.22.c). When block B2 output is selected as the sensor output,strip-shaped images of lines 4, 5 and 6 are read (FIG. 22.f).

[0159] Meanwhile, when the first and fourth lines are being output asblock outputs, the block B2 is selected to output line 4, and byswitching the multiplier 65 to select block B1, the output of lines 4, 2and 3 are successively provided as sensor outputs, and strip-images oflines 2, 3 and 4 are read (FIG. 22.d). When block B2 is selected forfirst two lines as the sensor output, lines 4 and 5 are output and thenblock B1 is selected and line 3 is output, then strip-shaped images oflines 3, 4 and 5 are read out (FIG. 22.e). In the figure, the referencecharacter ▾ represent a position of switching of the output from blockB2 to block B1. By switching the block selection signal during scanning,strip-shaped images at an arbitrary position having the same size asdivided block can be selectively read, though the order of output isdifferent.

[0160] The above described three different types of distance imagesensors allowing random access on row by row basis can be applied toreduce necessary input time to the three-dimensional shape measuringapparatus of the present embodiment.

[0161] The electronic circuit will be described. FIG. 23 is a blockdiagram showing the whole structure of the electronic circuit. The bodyof the measuring apparatus of the present embodiment is controlled bytwo microcomputers, that is, a microcomputer CPU1 controlling lighttransmitting and receiving systems lens driving circuits 71, 72, an AFcircuit 73, an electric universal head circuit 76 and input/output 75,74 and so on, and a microcomputer CPU2 controlling image sensor drivingcircuits 13 and 23, laser polygon driving circuits 77 and 78, a timer79, an SCSI controller 80, a memory controller MC, a pitch-shifted imageprocessing circuit 83 and so on. Under the control of microcomputer CPU1controlling the lens, input/output and so on, the power is turned,signals corresponding to key operation for sensingmode and so on arereceived from a control panel 75, and control signals are transmitted tomicrocomputer CPU2, light receiving system lens driving portion 71,light projecting system lens driving portion 72, AF driving portion 73,display image generating portion 74 and so on, so as to control zooming,focusing, sensingoperation and so on.

[0162] For color images, there are blocks of color image sensor 24,sensor driving circuit 23, analog pre-processing circuit 81 and imagememory 84. For distance images, there are blocks of distance imagesensor 12, sensor driving circuit 13, analog pre-processing circuit 82,pitch-shifted image processing circuit 83, and a pitch-shifted imagememory 85.

[0163] When the power is turned on, color image sensingsystem includingcolor image sensor 24, color image sensor driving circuit 23 and colorimage analog pre-processing circuit 81 are driven, and the photographedcolor images are displayed to the display image generating portion 74and displayed on a display 41 for the function of a monitor. Thesecircuits for color image sensingsystem are similar to the circuitsystems known in the conventional video camera or the like. Meanwhile,the sensors, lasers and so on for the distance image sensingareinitialized when the power is turned on, but they are not driven excepta polygon mirror driving circuit 78, which is driven at the time ofpower on since the time necessary for attaining normal speed of rotationof the mirror is relatively long. In this state, the user prepares forreleasing for image input, by setting the field of view by power zoomoperation, referring to the color image on the monitor display 41. Whenrelease operation is performed, a release signal is generated andtransmitted, so that the distance image sensingsystem including distanceimage sensor 12, distance image sensor driving circuit 13 and distanceimage analog pre-processing circuit 82 and laser driving circuit 77 aredriven, and image information is taken in pitch-shifted image memory 85and color image memory 84, respectively.

[0164] As for the color image, the information is supplied as analogsignals to the monitor apparatus. However, to color image frame memory85, the image input is provided as digital information, by A/Dconversion at an A/D converter AD1. These processes are similar to theknown technique in the field of digital video, digital steel video andso on.

[0165] As for the distance image, the microcomputer CPU2 waits for ascan start signal of the slit-shaped laser beam, transmitted from thescan start sensor 33 shown in FIG. 7. Thereafter, it waits for the deadtime Td for the unnecessary scanning derived from the distance d for themeasurement reference plane, base length 1 described above. After thedead time Td is counted from the scan start signal, distance imagesensor 12 and driving circuit 13 therefor are driven, and taking of datastarts. The timing operation is performed by timer 79.

[0166] When the driving of the sensor starts, a slit-shaped laser beamhaving its length in the horizontal direction starts scanning downwardfrom the uppermost portion of the light receiving system scanning scope.At the same time, image integration in the distance image sensor starts.When the slit shaped light scans with the amount of change of the anglecorresponding to one pixel of the distance image sensor by the movementof polygon mirror 7, then high speed vertical transfer from the imageintegrating portion to the accumulating portion takes place. Thereafter,distance image sensor driving portion 13 is controlled such that theimage of the uppermost row is taken at the center of the strip-shapedregion, and the image is read. Simultaneously with the completion of thevertical transfer from the image integrating portion to the accumulatingportion, successively reading process of the image from the accumulatingportion as the output of the distance image sensor, and chargeaccumulating process corresponding to the input light amount at theintegrating portion for the image reading of the next time, are carriedout.

[0167] When reading of one strip-shaped image is completed in thismanner, the slit-shaped laser beam again scan with the amount of changeof the angle corresponding to one pixel of the distance image sensor,and high speed vertical transfer from the image integrating portion tothe accumulating portion takes place. The strip-shaped region of thedistance image sensor is shifted downward by 1 pitch with respect to theregion which has been just read out, and image is read out.

[0168] By continuously repeating the series of operations, input ofstrip-shaped images is repeated successively, and 324 images areobtained. Since polygon mirror is kept rotating at a constant speedduring these operations, strip-shaped images corresponding to slitshaped lights having different light-section are input. The output fromthe distance image sensor is processed by distance image analogpre-processing circuit 82. More specifically, the output is subjected tocorrelative double sampling offset, processing of the output, and so on.Thereafter, the resulting output is converted to a digital signal by AIDconverter AD2, and transmitted as digital data to pitch-shifted imageprocessing circuit 83.

[0169] In pitch-shifted image processing circuit 83, calculation of thecentroid, that is, conversion from data of one strip-shaped image(including 256×16 pixels) to the position of centroid of the receivedlaser beam at 256 points is performed, using the received light beamcentroid calculating circuit (described later) shown in FIG. 24. Thecalculated amount of pitch-shift is stored in pitch-shifted image memory85. By repeating this operation for 324 times, 256×324 pitch-shiftimages can be obtained.

[0170] By the above described processing, images are stored inpitch-shifted image memory 85 and color image memory 84, respectively.These two images can be output as digital data to SCSI terminal 49 or tointernal MO 22 and so on, through memory controller MC under the controlof microcomputer CPU2 in charge of memory control, or output to LCDmonitor 41 and NTSC output terminals 50, 51 as NTSC signals by theconversion through D/A converter DA1.

[0171] When the output is to be provided from SCSI terminal 49, severalseconds are necessary to complete transmission of 1 set of output imagesof the color-pitch-shifted images when the output is in accordance withSCSI standard. Therefore, generally, the color images are recorded by avideo equipment as color NTSC signals generally used in a videoequipment, the pitch-shifted images are treated as luminance signals ofthe NTSC signals, and the monochrome image is output as a pitch-shiftedimage NTSC signal, whereby the color/pitch-shifted image as motionpicture can be output. When a high speed image processing apparatus isused, input of real time video image to a computer is possible.Alternatively, the NTSC signal may be connected to a common videoequipment and recorded, and thereafter the density images (pitch-shiftedimages) may be processed frame by frame during reproduction to be inputto the computer. By utilizing the color and pitch-shifted images of amoving object input to the computer, the present invention can also beapplied to the field of motion analysis of a moving object, for example.

[0172] Further, a rotary frame control portion 76 controlling panningand tilting operations of the electric universal head 4 on which themeasuring apparatus of the present invention is mounted may be providedas an external equipment of the system. The control operation using suchsystem will be described later.

[0173]FIG. 24 shows a detailed structure of the received light centroidcalculating circuit in the pitch-shifted image processing image 83. Thiscircuit has such a hardware structure that calculates the centroid basedon information at 5 points out of 16 points of data of a strip-shapedimage. Only effective pixels are extracted from signals from distanceimage sensor 12 by analog pre-processing circuit 82 and A/D converted byan A/D converter AD1, and the resulting signal is input through an inputterminal input at the left end of FIG. 14 to the circuit. The inputsignal is stored for 256×4 lines by 256×8 bits of FIFO (First In FirstOut) by using four registers 101 a to 101 d, and with the addition of 1line input directly, a total of 5 lines are used for calculation.Registers 103 a and 104 are the same as register 101, which is 256×8bits register. Register 109 is an FIFO register of 256×5 bits. Registers103, 104 and 109 are each provided in duplicate for the sameapplication, since larger memory capacity is preferred as time ofseveral pulses of the clock are necessary for the processings inselecting circuits 106, 108 and comparing circuit 107 and so on. Morespecifically, these two registers are alternately used, one for theodd-numbered data (O) and one for the even-numbered data (E), and whichof these should be used is controlled by clock pulses RCLK_0, RCLK_E.The centroid of the received laser beam is calculated based on data offive points of five lines, in accordance with the following equation.Since the intensity of received light become highest near the positionof the centroid, the point of the centroid at Ith row (1=1−256) iscalculated by obtaining n=N(I) where

Σ(I,n)=D(I,n+2)+D(I,n+1)+D(I,n)+D(I,n−1)+D(I,n−2)  (4)

[0174] becomes the maximum for each I. Assuming that there is thecentroid near N(I)th column, the amount of interpolation correspondingto the weighted mean Δ(I,N(I)) is calculated in accordance with thefollowing equation:

Δ(I,N(I))={2*D(I,N(I)+2)+D(I,N(I)+1)−D(I,N(I)−1)−2*D(I,N)(I)−2)}/Σ(I,N(I)).  (5)

[0175] Finally, the position of the centroid to be obtained is definedas

W(I)=N(I)+Δ(I,N(I))  (6)

[0176] where D (I,n) represents data at Ith row and nth column. Here, 1column includes 256 data, in register 101 a, data of D(I,n−1) is held,in register 101 b, data D(I,n) is held, in register 10 c, data D(I,n+1)is held, and in 101 d, data of D(I,n+2) is held, and these data are usedfor calculation. The calculation of Σ(I,n) (equation (4)) is performedby an adding circuit Σ, and the result is stored in register 104. Theresult of the next calculation is compared with the value MAX (Σ(I,n))which was calculated last time and stored in register 104 of each row(comparing circuit 107). If the present result is larger, the content ofregister 104 is updated, and the value of{2*D(I,n+2)+D(I,n+1)−D(I,n−1)−2*D(I,n−2)} calculated at the same time(=numerator of the equation (5)=R1) is updated and stored in register103, and the column number n is updated and stored in register 109. Asfor the calculation of R1, data D(I,n+2) and D(I,n−2) are shifted by 1bit to the left by a shift circuit 102, so as to realize the processingof (×2). Thereafter, calculation is performed by an adding circuit (+)and a subtracting circuit (−), and hence R1 is calculated at the pointA, which value is stored in register 103.

[0177] As for the column number n, the clock pulse PCLK is counted by a5 bit binary counter 110, and when the maximum value is updated as aresult of comparison at comparing circuit 107, the counter value at thattime is taken and stored in register 109. In this embodiment, the numbern is in the range of from 1 to 16. Therefore, 5 bits are sufficient forthe register 109 and binary counter 110.

[0178] By repeating this operation for one strip-shaped image, thevalues N(I), Σ(I,n(I)) and Y1 which provides MAX(Σ(I,n)) necessary forcalculation of the above equations are stored in registers 109, 104 and103, respectively. When Σ(I,N(I))=R2, then R1/R2 is calculated by adividing circuit (÷) and calculation of R1/R2+N(I)=Δ(I,N(I))+N(I)=W(I)is calculated by an adding circuit (+). Finally, the value of W(I) for256 columns is output from an output terminal output at the right end ofthe figure.

[0179] By storing 256 values of W(I) in pitch-shifted image memory 85and by repeating this processing for 324 strip-shaped images,pitch-shifted image consisting of pitch-shift information W(I) of256×324 points is formed on pitch-shifted image memory 85.

[0180] The operation of the apparatus will be described in detail withreference to flow charts. FIG. 25 is a flow chart of a main routineexecuted when the main switch is turned on. First, in step #1, devicessuch as CPUs, memories, SCSI, MO, display, control panel and so on areinitialized, in step #3, operation mode is determined. The operationmode includes a camera mode in which three-dimensional measurement iscarried out, and a replay mode in which the three-dimensional data isread from a storage device 22 such as MO and displayed in an internaldisplay, which modes can be selected by switch operation. Alternatively,either of the modes can be set as a default mode. In the replay mode,the flow proceeds to step #5 and processes for the replay mode, whichwill be described later, are executed. In the camera mode, the flowproceeds to step #6, and the processes for the camera mode, which willbe described later, are executed. When camera mode terminates, thepolygon mirror is stopped in step #7, and in step #8, AF/PZ is reset andthe lens is returned to the initial position. In step #9, the imagesensor and the sensor driving circuit are stopped, then the flow returnsto the step #3 for determining the operation mode.

[0181] The operation in the camera mode will be described with referenceto the flow chart of FIG. 26(a). When the camera mode is selected, instep #11, various devices are initialized, in step #13, the color imagesensor is activated, and the color image is supplied to a monitordisplay 41. As for the image, an auto focus sensor 31 arranged in thelight receiving zoom lens is driven so that the light is always receivedwith optimal state of focusing and optimal color image is obtained.Next, in step #15, driving of the polygon mirror which requires longtime to reach the stable state is started earlier so as to be ready forsensing of the distance image. In step #17, AE/PZ subroutine isexecuted. In step #19, the flow waits until the operation of the polygonmirror becomes stable. When it becomes stable, the flow enters theshutter mode at step #21, and the shutter mode subroutine is executed.In step #23, data transfer mode starts and the data transfer modesubroutine is executed. In step #25, whether the camera mode iscompleted or not is determined, and if it is completed, the flowproceeds to step #27 and returns to the main flow. If not completed, theflow returns to step #21.

[0182] The AF/PZ subroutine will be described with reference to FIG.26(b). In step (#31) the lens position of the light projecting andreceiving systems are reset, in step #33, the range of laser scanning isreset, and in step #35, the flow returns to the main flow.

[0183] The shutter mode operation will be described with reference tothe flow chart of FIG. 27. In this state, the user switches the trimmingby changing the position of the measuring apparatus, attitude and thezooming magnification. Meanwhile, the apparatus waits for an output of arelease signal by the pressing of shutter release button 47. In step#41, an AF/AE subroutine is executed in which the apparatus is set toin-focus state, and the brightness is measured. This subroutine will bedescribed later. In step #43, the states of select key 43 and AF/AE arechecked. First, whether the select key has been pressed or not isdetermined. If it has been pressed ([Select]), the flow returns to themain flow in step #79 so as to go out of the shutter mode. Thiscorresponds to release of the first stroke of the shutter release buttonin a general single lens reflex camera. If the select key has not yetbeen depressed, the state of AF/AE processing is checked and if theAF/AE processing is being carried out, the flow returns to #41, andAF/AE processing is repeated. If AF/AE processing has been completed,the flow proceeds to the next step (#45). More specifically, in theprocessing period described above, focusing and measurement ofbrightness for the light receiving system and light projecting systemzoom lenses are repeated continuously, so that the in-focus state isalways maintained.

[0184] After the completion of AF/AE, lock of the shutter button 47 isreleased to be ready for sensing in step #47, and driving of focusing orzooming is inhibited (AF/PZ lock). In step #47, whether the shutterbutton 47 is pressed or not is determined. If the shutter button isdepressed, the flow proceeds to step #55. If not, the flow proceeds tostep #51 in which whether a prescribed time period has passed or not isdetermined. If the prescribed time period has not yet lapsed, the flowreturns to step #47 and determines whether or not the shutter button ispressed. If the prescribed time period has passed, the flow proceeds tostep #53 in which operation of the shutter button is locked and then theflow returns to step #41.

[0185] In step #55, laser beam is projected, and in step #57, the flowwaits for the rise of the laser beam until it reaches the normaloscillation and also waits for the completion of the preparation ofpolygon mirror operation. When the preparation is completed, driving ofthe sensor is started in step #59. In step #61, the flow waits until anoutput from scanning start sensor is received, which sensor is attachedaside the condenser lens. When the scanning start signal is received, instep #63, the flow waits for the dead time Td, and then starts drivingof the distance image sensor. The dead time Td is calculated based onthe focal length f, the base length 1 and the distance d to thereference plane for measurement. In step #65, the position of the inputstrip-shaped image of the distance image sensor is set to the initialposition, and operation for taking the pitch-shifted image and colorimage is started. At the same time, the position of the centroid of theinput light is calculated. In step #67, whether the scanning has beencompleted or not is determined. If not, the flow returns to step #65 andrepeats taking of images. With the position of the input strip-shapedimage shifted pitch by pitch from the initial position in accordancewith the scanning by the slit-shaped laser beam, 324 strip-shaped imagesare taken.

[0186] When driving of the sensor is started, subsequent to the start ofdriving the distance image sensor, the color image sensor is drivenagain in step #69, and in step #71, the read color image is taken incolor image memory. Driving of both image sensors and taking of theimages to the memories are adapted to be performed simultaneously andautomatically by hardware structure. Then the flow proceeds to step #73.After the completion of taking of the pitch-shifted images and the colorimages, emission of the laser beam is stopped in step #73, inhibition ofzoom driving and focus driving is canceled in step #75, the taken imageis displayed in accordance with the selected mode in step #77, and theflow returns to the main flow in step #79.

[0187] The flow chart of the AF/AE subroutine of step #41 will bedescribed with reference to FIG. 28. First, in step #91, the amount ofdriving the lens is calculated based on the information from AF sensor31, and based on the result of calculation, the focusing lens is driven(step #93). In step #95, the scan start laser position is set, and laserpower is controlled in step #97. In step #99, brightness is measured(AE), and the flow returns to the main flow in step #101.

[0188] The data transfer mode will be described with reference to theflow chart of FIG. 29. First, in step #111, display mode is determined.More specifically, the flag is checked so as to determine whether thepitch-shifted image in which the image is displayed in light and shadeor a color image, is selected. The display mode can be selected by keyoperation and color image display is set in the default state, forexample. When there is no key operation or when color image is selectedby the key operation, a color image is displayed in step #113. When thedisplay of the pitch-shifted image is selected, the pitch-shifted imageis displayed in step #115. After the image display, in step #117,whether the display mode is changed or not is determined. If it ischanged, the flow returns to step #111 and provides image display inaccordance with the selected mode. If the display mode is not changed,then the flow proceeds to step #119.

[0189] In step #119, whether data transfer is necessary or not isdetermined. If data transfer is not necessary, the flow proceeds to step#133, in which color image is displayed. When data transfer isnecessary, then data header is provided in step #121. In step #123,whether it is an SCSI output mode is determined if the SCSI output modeis selected, in step #125, data for external output is provided and instep #131, data transfer is carried out. If it is not the SCSI outputmode, it means recording by an internal recording apparatus. Therefore,in step #127, data for internal MO drive is prepared, in step #129, datatransfer instruction to the MO is transmitted from CPU2 to SCSIcontroller, and in step #131, data is transferred. Thereafter, in step#133, color image is displayed, and in step #135, the flow returns tothe main flow. Selection of the data transfer destination can beselected by key operation.

[0190] The replay mode will be described. In step #3, a switch forswitching to the replay mode is checked in step #3. If it is notswitched, the flow enters a standby state (camera mode) for the nextimage input, and if it is switched, then the flow enters the replaymode.

[0191] The replay mode is different from the camera mode describedabove, and in this mode, image data which has been already recorded inthe internal recording apparatus such as MO is replayed forre-confirmation, or the recorded image data is output to an externalapparatus through SCSI terminal, for example. The operation in thereplay mode will be described with reference to the flow chart of-FIG.30.

[0192] First, in step #151, a list of images stored in the MO isdisplayed. In step #153, the user selects a re-confirmation display orimage data to be transferred to the external apparatus from the displayof the list. In the next step #155, the color/pitch-shifted image as theselected image data are loaded from the internal MO to colorimage/pitch-shifted image memories 84, 85, respectively. In step #157,whether the image to be displayed is in the color image display mode orpitch-shifted image display mode is determined by checking the flag. Inaccordance with the selected display mode, in step #159, the color imageis displayed, or in step #161, the pitch-shifted image is displayed.After the display of the image, in step #163, whether the display modehas been changed or not is determined. If it is changed, the flowreturns to step #157 and again provides a display.

[0193] If it is not changed, whether the next image data is to bedisplayed is determined in step #165. If the display is desired, theflow returns to the first step #151 of the subroutine, and repeatsselection and display of the images. When the next image is not to bedisplayed, then in step #167, whether the image data read from the MO tothe memory is to be externally transferred is determined. If it is notto be transferred, the flow jumps to the step #173. If the data is to betransferred, in step #169, data for external output is provided, and instep #171, data transfer is performed. Then, in step #173 whether thenext image data is to be displayed or not is determined. If display isdesired, the flow returns to step #151, and if not, the flow returns tothe main flow in step #175.

[0194]FIG. 31 shows state transitions by key operations between theabove described series of operations.

[0195] Referring to the figure, the sign A pointing upward, downward,left and right directions denotes the operation of a cursor key 42 ofFIG. 3. The reference characters [Shutter], [Select] and [Cancel] denoteoperations of shutter button 47, select key 43 and cancel key 44,respectively. Though clock function is provided in the presentembodiment, the time may be automatically allotted to the file name ofthe image file which is to be recorded.

[0196] Reproduction display, list display and clock function can beselected on the menu display, and any of these can be selected andexecuted by operating the cursor key in the left and right directionsand the select key 43. After the selection and execution, the firststate which allows selection of the menu display is restored by theoperation of cancel key 44. By the clock function, the time can be set.List display function allows file operations such as change of filename, erasure, selection of the file to be displayed, and so on. As tothe reproduction display function, color image display is set as thedefault state. By moving the cursor key in the left and rightdirections, the display can be switched to the pitch-shifted imagedisplay and character image display. In respective display modes,previous image and succeeding image can be displayed by operating thecursor key in upward and downward directions. In the character display,file operations such as change of the file name and erasure can becarried out by the select key.

[0197] When a cancel key is operated on the menu display, the operationenters the sensing stand-by mode (camera mode), and operation of theselect key restores the menu. When the shutter button is pressed in thesensing stand-by state, sensing is possible and image is taken to thememory. After the sensingoperation, by operating the cancel key, theoperation can be returned to the sensing stand-by state. When the selectkey is operated immediately after the sensingoperation, recording of theimage can be done, and the image taken in the memory is transferred tothe memory device. After recording, the operation returns to the sensingstand-by state. At this time, color image is set as the default state,and pitch-shifted image/color image can be selected by the operation ofthe cursor key in the left and right directions. It is possible toconvert the pitch-shifted image to the range image in the measuringsystem without using the outside computer system, and to display therange image as the density image.

[0198] Next, highly precise input by divisional taking by thethree-dimensional shape measuring apparatus will be described. When thedistance between the light projecting system and the light receivingsystem, that is, base length 1, focal length f and distance d to theobject to be measured are determined, three-dimensional resolution andprecision are determined. Measurement with high precision is attained bymeasuring with the focal length f set at a large value. In other words,the precision in measurement increases in teleside. However, though athree-dimensional image with high precision for measurement can beobtained, the field of view becomes narrower as the focal length fbecome longer.

[0199] Therefore, the focal length f is set to a value corresponding tothe desired resolution and precision for measurement and the range ofthe field of view is divided into a plurality of regions by operating arotary frame 4 such as an electrical universal head. Measurement isperformed for every divided region, and the resulting images are puttogether or patched up to re-construct one image. By providing such afunction, a three-dimensional shape measuring apparatus of whichresolution can be varied is realized. By utilizing this function,environmental measurement becomes possible by performingthree-dimensional measurement of the entire peripheral space. Thisoperation will be described referring to a specific example. The exampleshown in FIG. 32 is a simplified illustration, in which the lightprojecting system 2 and light receiving system 3 are arranged atpositions having horizontal relation, which is different from theexample shown in FIG. 3. In this arrangement, the slit shaped light hasits length extending in the longitudinal direction, and thereforescanning must be carried out in left and right directions.

[0200] The manner of operation utilizing the image patch up function isshown in FIG. 32. FIG. 33 is a flow chart of the operation utilizing theimage patch up function. FIG. 34 shows the state of display when thisfunction is used, in which there is provided a display portionindicating the precision in measurement below the image display portion.

[0201] First, referring to FIG. 32(a), the zoom drive system 16 isdriven to set the range of the field of view (step #201) to a wide anglestate (focal length f0)., allowing sensing of the object 1 in the rangeof the field of view, by the operation of the user. The resolution inthe Z axis direction (see FIG. 17: the direction of the ups and downs ofthe object) assumed at this time is represented by a bar indicationbelow the image, as shown in FIG. 34(a). When the base length is fixedas in the present system, briefly, the resolution ΔZ in the direction ofthe Z axis satisfies the following relation between the distance d tothe object to be measured and the focal length f at the time ofmeasurement:

ΔZ=K×d(d−f)/f  (7)

[0202] where K is a coefficient for estimating the resolution in thedirection of the Z axis, which is determined by the sensor pitch and soon. The zooming operation described above is performed by transmitting acommand from a system computer through SCSI terminal. Setting ofoperations such as zooming operation and releasing operation can be setby remote control.

[0203] When the user determines that the above described setting allowsmeasurement with sufficient precision and sufficient resolution (NO instep #203), then measurement is started by the releasing operation bythe user (step #205), and the result is given on the display (step#207). In this display, the input pitch-shifted image or color image isdisplayed, as well as the measurement resolution in the direction of theZ axis obtained at that time, displayed in the shape of a bar below theimage, as shown in FIG. 34(a). As a result, if measurement with higherprecision is not necessary (NO in step #209), measurement is completed,and whether or not the obtained result is to be written to a storagemedium is determined, and the corresponding processing is performed.Thus the operation is completed.

[0204] When the user determines that measurement is not performed withsufficient precision (YES in step #203), the user can instructre-measurement with the precision and resolution changed by keyoperation to the desired resolution in the direction of Z axis anddesired precision, referring to the pitch-shifted image taken by thefirst releasing operation or referring to the display of the measuredresolution in the direction of the Z axis (YES in step #209).

[0205] When the key input for setting the precision is entered, thesystem stores the state at that time. More specifically, the systemstores the focal length f0 at which the complete view of the object isobtained, and approximate distance d to the object to be measuredobtained from the AF sensor, and hence stores the scope of the field ofview (step #210). Further, based on the input desired measurementresolution in the direction of the Z axis and the approximate distanced, the system calculates the focal length f1 to be set in accordancewith the equation (7) above (step #211).

[0206] When the focal length f1 is calculated, automatic zooming isperformed to the focal length f1; (step (213); the number of frames tobe input in divided manner and the angles of panning and tilting arecalculated based on the stored scope of the field of view to bemeasured, the approximate distance d, and the focal length f1; theposition of the field of view is set by panning and tilting rotary frame(step #215); and measurement is performed for every divided input frame(step #217). The images input dividedly when the image patch up functionis utilized are set to include overlapping portions which are used forpatching up the divided images to re-construct the original one image.

[0207] The obtained pitch-shifted image, color image, informationindicative of the directions of the field of view in the X and Ydirections taken (for example, decoded angle values of panning andtilting, order of taking in the X and Y directions, and so on), the lensfocal length, information of measurement distance are stored in aninternal MO storage device (step #219). At this time, directoryinformation such as file name, file size and so on may not be written tothe memory but such directory information may be written afterconfirmation by the user at the last step of operation, so that theinformation is stored temporarily.

[0208] Thereafter, by controlling the field of view to a position of thefield of view slightly overlapping the position of the field of view ofthe previous operation by panning and tilting in accordance with thecalculated angles of panning and tilting, the image of the adjacentregion is input. By repeating this operation, the images of the entireregions are input (NO in step #221, see FIG. 32(b)).

[0209] At the completion of the input of the entire region (YES in step#221), the initial camera attitude and initial focal length beforeenhancing the precision in measurement are resumed (step #223) and hencethe operation is completed. The control waits for the determination ofwriting by the user. When there is a write instruction, directoryinformation is written. If there is not a write instruction, thedirectory information is not written and the operation is completed. Inthat case, the information continuously stored in the memory is erased.

[0210] When a measurement is performed in advance and thereaftermeasurement is again performed as in the operation of the above example,the distance to the object and distribution of the distance in the viewangle of measurement have been completed by the first measurement.Therefore, re-measurement for patching up is not performed for such adivisional input frame having large difference from the distance to theobject, that is, the frame consisting only of the peripheral region(background) different from the object, and re-measurement may beperformed only for the divided input frames including the object to bemeasured. In the example shown in FIG. 34(b), the dotted regionincluding the object of measurement corresponds to the region for whichre-measurement is performed. Other regions do not include the object formeasurement and therefore re-measurement is not performed.

[0211] As described above, high speed three-dimensional measurement ispossible, and by repeating partial inputs and patching up the resultingimages based on the three-dimensional measurement, three-dimensionalshape measurement can be performed of which resolution can be setfreely.

[0212] In such a patch up measurement, the resolution of the whole imageframes are uniform. However, there may be an object which require datawith high resolution at some portion but low resolution for otherportions. For example, eyes, mouth and nose of one's face are abound incomplex shape and color information, while low resolution is sufficientfor measuring cheeks, forehead and so on. For such an object, patch upof data may be utilized by partial zooming operation, which results inhighly efficient data input. The partial zooming patch up function isrealized by the following operations.

[0213]FIG. 35 is a flow chart showing the partial zooming patch upfunction. First, in step #251, setting of the field of view providingthe complete view of the object is performed, in the similar manner asthe uniform resolution patch up described above. In step #253, partialzooming input mode is selected. When selection is done, presently setvalues of focal length f0 and values of decoded angles of panning andtilting are stored (step #255). Measurement is started with focal lengthf0, and image input is provided as rough image data (step #257). Thepitch-shifted image, color image, information indicating the directionsof the field of view in the X and Y directions at which the image istaken (for example, decoded angle values of panning and tilting), thelens focal length, and information of -measurement distance are storedin an inner storage device (step #259). Thereafter, in step #261,zooming is performed to attain the maximum focal length fmax, the roughimage data mentioned above is analyzed, and whether or notre-measurement is to be performed on every divided input frames inputafter zooming is determined.

[0214] When zooming is performed and measurement is done with themaximum focal length fmax, the approximate data is divided to the framesize which allows input. The positions X, Y for panning and tilting areset to the start initial positions Xs and Ys. In step #265, panning andtilting are controlled to the positions X and Y. Then, in step #267,color information, i.e., R. G and B values of the initial input colorimage of the region X±ΔX and Y±ΔY are subjected to statisticalprocessing, and standard deviations σR, σG and σB of respective regionsare calculated. In step #269, whether all the calculated values of thestandard deviations σR, σG and σB are within the set previous values aredetermined. If these are within the prescribed values, it is determinedthat the small area has uniform brightness information, and thereforezooming measurement is not performed but the flow proceeds to step #271.When any of the standard deviations. σR, σG and σB exceeds theprescribed value, it is determined that the small region has complicatedcolor information, and therefore zooming measurement is performed (step#275).

[0215] In step #271, standard deviation σd is calculated based on theinformation of the initial input distance value d in the region of X±ΔX,Y±ΔY. In step #273, whether the calculated value of the standarddeviation ad is within a set prescribed value is determined. If it iswithin the prescribed value, it is determined that the small region is aflat region having little variation in shape, and therefore zoomingmeasurement is not performed but the flow proceeds to step #279. If itexceeds the prescribed value, it is determined that the small region hascomplicated shape (distance information), and zooming measurement isperformed (step #275).

[0216] After the zooming measurement in step #275, the obtainedpitch-shifted image, color image, information indicative of thedirection of the field of view of Z and Y directions at which the imageis taken (for example, decoded angle values of panning and tilting),lens focal length, information of distance for measurement and so on arestored in an internal storage device such as MO (step #277). Thereafter,the flow proceeds to step #279.

[0217] In step #279, the panning and tilting position X is changed by2ΔX. In step #281, whether or not scanning in the X direction iscompleted is determined. If it is not completed, the flow returns to thestep #265. If it is completed, the panning-tilting position Y is changedby 2ΔY in step #283. In step #285, whether scanning is completed or notis determined, and if it is not completed, the flow returns to step#265. If the scanning is completed, the flow proceeds to step #287, andthis routine terminates.

[0218] In this manner, both the schematic image data and partialdetailed image information allowing determination of the position can beinput. By patching up the data of the schematic image and the partialdetailed image data corresponding to the position, highly efficientthree-dimensional input corresponding to how complicated the shape andcolor information can be realized.

[0219] The second embodiment of the present invention will be described.Patching up when sensingoperation is done with the camera mounted on anuniversal head will be described. The patched up image is obtained bytaking a plurality of pictures of a fixed object by panning and tiltingthe camera mounted on the universal head which allows panning andtilting, and the data of the plurality of photographed images areconverted to one coordinate system to obtain the patched up image.

[0220] When the camera is to be panned and tilted, highly precisepatching up is possible -without any problem if the angle of rotationcan be controlled precisely. However, highly precise universal head isvery expensive, and therefore sensing by using general, not so expensiveuniversal head is desired, which universal head may have considerableerror in controlling the angle of rotation.

[0221] In that case, a camera model such as shown in FIG. 36 isprepared. This is a camera which allows panning and tilting, representedin three-dimensional coordinate system. In FIG. 36, the referencecharacter C denotes the camera, e denotes the axis of rotation of thecamera (panning), and Φ represents the axis of rotation of the camera(tilting).

[0222] Parameters of the model (position and direction of the axis ofrotation for panning, position and direction of the axis of rotation fortilting) are calculated in advance by calibration. Searching of thejunction point (at which two image data are jointed) carried outsubsequently is performed by changing parameters θ (pan angle) and Φ(tilting angle) of the model.

[0223] This operation will be described with reference to the flow chartof FIG. 37. First, two-dimensional color image, three-dimensional data,focal length and the distance to the reference plane are taken from thephotographed data (stored in the storage device in the camera apparatus,as described above) (step #301). Then, the junction point is searchedfrom the two-dimensional color image (step #302: details will be givenlater).

[0224] However, the points of measurement of the two images photographedby panning and tilting the camera do not always coincide with each other(even when the images are photographed with the same sensing distanceand focal length, the images deviate from each other by half pixel, atmost). Therefore, when the deviation is within 1 pixel, searching of thejunction point is regarded successful, based on the color image(two-dimensional data), and searching hereafter is performed by usingthe three-dimensional data.

[0225] First, based on the junction point of the two-dimensional images,the focal length and the distance to the reference plane, the angles ofcamera rotation (pan angle θ and tilting angle Φ) are calculated (step#303, details will be given later). Thereafter, according to thecalculated camera rotation angles, coordinate conversion parameters forthe three-dimensional space are calculated (step #304, details will begiven later).

[0226] Then, a square sum of an angle formed by normals of two planespassing through the Junction portion is regarded as an evaluation value,and search for minimizing the evaluation value is performed (step #305).The method of calculating the evaluation value will be described indetail later.

[0227] Since rough search is performed using the two-dimensional images,search for a very narrow scope is enough for the three-dimensional data.Therefore, the amount of calculation in total can be reduced as comparedwith the search utilizing the three-dimensional data only, whereby patchup operation can be done at high speed.

[0228] Thereafter, whether the calculated evaluation value is within aprescribed value is determined (step #306). When it is not larger thanthe prescribed value, two three-dimensional images are converted to thesame coordinate system by using the last calculated coordinateconversion parameter for patching up (step #307) and the operation iscompleted (step #310). The method of coordinate conversion will bedescribed later.

[0229] When the evaluation value is larger than the prescribed value instep #306, then whether the number of repetition of continuityevaluation is not smaller than a prescribed number is determined (step#308). If it is not smaller than the number, the patch up operation instep #307 is performed. The reason is that when the number of repetitionexceeds a prescribed number, the evaluation value converges, makingfurther repetition unnecessary.

[0230] When the number of repetition is smaller than the prescribednumber in step #308, then the angle of rotation of the camera isslightly changed so that the evaluation value calculated in step #305becomes smaller (step #309). Thereafter, the flow proceeds to step #304and continuity evaluation of the patches at the junction portion isrepeated.

[0231] In the following, each of the steps will be described in greaterdetail.

[0232] First, the method of searching the junction point fromtwo-dimensional color images in step #302 will be described withreference to FIGS. 38 to 40. The description will be given on thepremise that two images to be patched up have overlapping portions(having the width of T pixels) as shown in FIG. 38. Referring to FIG.39(a), a reference window is set at a central portion of the overlappingportion of one of the images (the dotted line in FIG. 39(a) denotes thecenter line of the overlapping portion). FIG. 39(b) is an enlarged viewof the reference window portion of FIG. 39(a). This reference window isfurther divided into small windows each having the size of about 8×8(pixels). Of the small windows, one having a complicated shape orcomplicated patterns (having large value of distribution) is used as acomparing window. The reason for this is that when a portion havingclear edges or complicated patterns or shapes is used, reliability ofevaluation can be improved.

[0233] On the other image, a searching window which has the same size asthe reference window and which is movable to move on the entireoverlapping portion is set (FIG. 40).

[0234] In this searching window, small windows are provided atrelatively the same positions as the comparing windows in the referencewindow. The square sum of the difference in luminance between the smallwindow and the comparing window is used as the evaluation value, and thejunction point is searched.

[0235] Next, the method of calculating camera rotation angle based onthe two-dimensional image junction point, the focal length and thedistance to the reference plane of step #303 will be described. When werepresent the pixel size as PS, camera plane size as 2×S, focal length fand the number of shifted pixels as T, the camera rotation angle θ canbe obtained by the following equation if the axes of rotation and thecamera position coincide with each other (that is, the rotation axisintersects the optical axis of the camera) (FIG. 41):

θ=π−arctan(S/f)−arctan((S−PS×t)/f).

[0236] If the rotation axis does not coincide with the camera position(when the rotation axis and the camera axis are deviated from eachother), the following relation holds, where r represents radius ofrotation (distance between the rotation axis and the optical axis of thecamera), and D represents the distance to the reference plane:

t×PS×D/f=2S×D/f−(D+r×sin θ)/tan(π−arctan(f/S)−θ)−S×D/f−r×cos θ.

[0237] When the rotation axis and the camera position do not coincidewith each other, the calculation becomes very complicated and the angleof rotation cannot be obtained easily. Therefore, it is preferable toprovide a table showing number of pixels (t) and corresponding anglesobtained by searching, so that the angle of rotation can be readilyfound.

[0238] The method of calculating the coordinate conversion parameter ofthe camera in step #304, and the method of coordinate conversion of step#307 will be described. when we represent the coordinate systems of twocameras as C1 (X1, Y1, Z1) and C2 (X2, Y2, Z2), the position of thecamera rotation axis as T (t1, t2, t3) and the direction of rotation ofthe camera as (1, 0, 0) (rotation about the X axis in the coordinatesystem of C1), then C2 when rotated by e about the axis of rotation isconverted to the coordinate system of C1 in accordance with thefollowing equation:

C2−T=R(θ)·(C1−T)

[0239] where R (θ) is obtained by the following equation, based on theangle e of camera rotation: ${R(\theta)} = {\begin{matrix}1 & 0 & 0 \\0 & {\cos \quad \theta} & {{- \sin}\quad \theta} \\0 & {\sin \quad \theta} & {\cos \quad \theta}\end{matrix}}$

[0240] Therefore, conversion of C2 coordinate system to the C1coordinate system can be represented by the following equation, usingparameters R (θ) and T:

C1=R(θ)⁻¹·(C2−T)+T.

[0241] More specifically, the point C1 (of C1 coordinate system) ismoved in parallel onto the rotation axis, the coordinate is converted tothe C2 coordinate system on the rotation axis (rotated by θ), and thepoint is moved in parallel from the rotation axis to the point C2.

[0242] The above described operation is for the panning angle. Similarcoordinate conversion can be performed by using the Y axis as therotation axis for the tilting angle.

[0243] The method of calculating the continuity evaluation value of thepatches at the junction portion in step #305 will be described ingreater detail with reference to the flow chart of FIG. 42 and FIG. 43.

[0244] First, three-dimensional data of two images including the pointof junction searched from the two-dimensional color images are taken(step #401). Thereafter, normals at the center of the planes 1 to 12 atthe portion of junction between the first image (the image representedby the white circle in FIG. 43) and the second image (the imagerepresented by the block circle in FIG. 43) are calculated (step #402).

[0245] Then, for the first image,

e1(1)=(angle provided by the normals of 1 and 2−1)−angle formed by thenovels of 1 and 2−12)

[0246] is calculated. Similar calculation is performed for n sets ofplanes following the plane 4, and square sum (e1) of the result isobtained (step #403). For the second image,

e2(1)=(angle formed by normals of 3 and 2−2)−(angle formed by normals of3 and 2−12)

[0247] is calculated, similar calculation is performed for n sets ofplanes following the plane 6, and the square sum (e2) of the results isobtained (step #404).

[0248] Then, whether a smooth junction is obtained or not is evaluatedby using

(E1+E2)/n

[0249] in (step #405), and the evaluation value is returned to the mainroutine (step #406).

[0250] Patched up of the data photographed by a plurality of cameraswill be described. When a plurality of cameras are used forphotographing, the relative position and orientation can be measured bysensing cameras by each other. Therefore, based on the data, theposition of the object (corresponding to the position of the rotationaxis) and the angle between the cameras viewed from the object(corresponding to the angle of rotation) are calculated. Based on thesecalculated values, coordinate conversion parameters are calculated, andtwo three-dimensional images are converted to the same coordinate systemand patched up. The details of the patching up operation is similar tothat when the camera frame is used described above. Therefore,description is not repeated.

[0251] When the cameras are photographed by each other, the cameraposition can be calculated with higher precision if a lens having longerfocal length than used for sensing the object is used. By doing so,undesirable influence on the object data at the time of patching can beavoided.

[0252] Next, patch up of data when the object is photographed placed ona rotary stage will be described with reference to FIG. 44 and the flowchart of FIG. 45. FIG. 44 shows a rotary stage on which the object isplaced. The rotary stage has polygonal circumference. The normal of eachplane is orthogonal to the rotation axis, and the planes are arranged atequal distance from the rotary axis. Therefore, when each plane ismeasured, the distance and position of the rotary axis of the rotarystage can be calculated.

[0253] For example, four three-dimensional and two-dimensional data arephotographed (step #502) by rotating the rotary stage by 90° for everysensingoperation, such that the rotary stage is within the measurementscope as shown in FIG. 46 (which is a model of sensingoperation usingthe rotary stage). Thereafter, with respect to the data of thephotographed four images, the data of the object and the data of therotary stage portion are separated from each other, and a group ofplanes which is lower part of the rotary stage is extracted (step #503).At this time, when the plane portion of the rotary stage may have aspecific color, so as to facilitate extraction with reference to thecolor image.

[0254] Thereafter, of the data separated in step #503, using the data ofthe rotary stage portion the position and attitude of the rotary stageare calculated (step #504). This method will be described in greaterdetail later.

[0255] Based on the position and attitude of the rotary stage and theangle of rotation calculated in the subroutine of step #504, coordinateconversion parameter (about the rotation axis) for each photographeddata is calculated (step #505). Based on the parameter, coordinateconversion is performed, whereby respective photographed data areintegrated in one coordinate system (step #506). The method ofcalculating the parameter from the rotation angle and the method ofcoordinate conversion are the same as the method of calculating theparameter and method of coordinate conversion when the above-describedcamera universal head is used, except that the angle of rotation of thecamera is replaced by the angle of rotation of the rotary stage.

[0256] Thereafter, the junction portion is set (the method will bedescribed later), data out of the scope of each photographed data isdeleted, a plane is re-constructed at the junction portion (step #507),and patching up of the three-dimensional data is completed (step #508).

[0257] As a result, the first image (represented by the black points)and the second image (white points) are patched up at the boundary, thusresulting in one image.

[0258] The method of calculating the position and the attitude of therotary stage in step #504 will be described with reference to the flowchart of FIG. 48. First, three-dimensional data of the rotary stage andthe color image of the rotary stage are taken (step #601). The data isdivided for each plane (step #602). Thereafter, normal vector of theplane is calculated for every plane (step #603). A line which isorthogonal to the normal vector and at an equal distance from respectiveplane is defined as the rotation axis (step #604), and the rotation axisis returned to the main routine as the position and attitude of therotary stage (step #605).

[0259] The method of setting the junction portion will be described ingreater detail.

[0260] First, an example, in which the real data photographed is notchanged, will be described with reference to the flow chart of FIG. 49.First, of the data cut at the boundary of the images (four planesincluding the axis of the rotary stage and orthogonal to each other),only that data which is sandwiched by two boundaries of images isregarded as effective data, and other data are canceled (step #701).Correspondence between end points of the photographed data is determined(two points which are close to each other are regarded as correspondingpoints), the images are patched up successively (step #702), and theflow returns to the main routine (step #703).

[0261] In this case, when the data are canceled, overlapping portionsmay be left for two images to be patched up. By doing so, patching upcan be performed smooth by searching the junction point, as alreadydescribed with reference to the patching up operation using a universalhead for the camera.

[0262] An example in which photographed data are changed for smoothpatch up operation will be described with reference to the flow chart ofFIG. 50 and to FIG. 51. With respect to photographed data, pointsapproximately at the same distance from the boundary between the images(four planes including the axis of the rotary stage and orthogonal toeach other) are determined as corresponding points (1-1 and 2-1, 1-2 and2-2, . . . , 1-n and 2-n of FIG. 51) (step #801), and a new point (thepoint marked by x in FIG. 51) is generated based on two pointscorresponding to the data up to a prescribed distance from the boundary(step #803). The new point which is to be generated is determined in thefollowing manner, in accordance with the distance from the boundary.

[0263] When we represent a prescribed scope (in which the new point isgenerated) from the boundary as D, the point of one photographed data asX1, the point of another photographed data as X2, average distance fromthe boundary to the two points (X1, X2) as d and the newly generatedpoint as X3, the following relation holds:

X3=((D+d)×X1+(D−d)×X2)/(2×D).

[0264] The plane is reconstructed by using the newly generated data nearthe boundary (the scope whose distance from the boundary is up to D) andby using real data at other portions (the scope whose distance from theboundary exceeds D) (step #803), and the flow returns to the mainroutine (step #804).

[0265] Further, by applying a recess/projection at every 90° on therotary stage as shown in FIG. 44(b), the angle of rotation can be madevery precise at extremely low cost. By performing coordinate conversionbased on the axis of rotation calculated in advance for the four imagesphotographed with the stage rotated, the entire peripheral data can beobtained. When such a rotary stage is used, the object can be set at anarbitrary position within the scope of measurement, and thereforesensingoperation is much facilitated.

[0266] Zoom patch up method when zooming input is provided as mentionedabove will be described with reference to the flow chart of FIG. 52.

[0267] First, photographed data having different magnifications aretaken in accordance with the method described with reference to zoominginput (step #901). Then, patched up of the images using the camerauniversal head is performed in the same manner as shown in the flowchart of FIG. 37 described above (except the last patch up operation),and parameters for coordinate conversion and extraction of data at theboundary portion are performed (step #902). Here, before searching thejunction point from two-dimensional color images (step#302 of FIG. 37),re-sampling is performed for the two-dimensional images and for thethree-dimensional images (the method will be described later).

[0268] Thereafter, coordinate conversion is performed with respect tothe data having high magnification, the data. having high magnificationis integrated to the coordinate system of data having low magnification(step #903), a plane is re-constructed at the boundary portion (step#904), and the patch up operation is completed (step #905).

[0269]FIG. 53 is a model of the zooming patch up operation. First, data(having magnification of N2) of FIG. 53(b) is re-sampled so that itcomes to have the magnification of N1 as shown in FIG. 53(c), and thenre-sampled data is patched up with the data (having the magnification ofN1) of FIG. 53(a). Thereafter, the portion which had the magnificationof N2 is returned to have the original magnification (N2), and as aresult, a patched up image such as shown in FIG. 53(d) is obtained.

[0270] The method of re-sampling of two dimensional image and thethree-dimensional image will be described in greater detail.

[0271] The method for the two-dimensional image will be described withreference to FIG. 54.

[0272] In FIG. 54, the image represented by the solid lines is the imagehaving the magnification of N1, while the image represented by thedotted lines is the image having the magnification of N2 (in bothimages, the minimum square corresponds to one pixel, where N1<N2).

[0273] Re-sampling is performed for the image having the magnificationof N2. The phase is matched so that the pixel at the upper left end ofthe image having the magnification N2 coincides with a sampling point ofthe image having the magnification of N1.

[0274] Re-sampling value (average brightness) is calculated by using aweighted mean value of the area of the pixels of the image havingmagnification of N2 included in the pixels of the image having themagnification of N1. More specifically, the product of the brightnessand area of the image having the magnification of N2 included in 1 pixelof the image having the magnification N1 are all added, and the resultis divided by the area of one pixel of the image having themagnification of N1.

[0275] The operation for the three-dimensional image will be describedwith reference to FIG. 55. FIG. 55 is a representation viewed from thecamera.

[0276] In FIG. 55, the image represented by the solid lines and thewhite circles is the image having the magnification of N1, while theimage represented by the dotted lines and the black circles is the imagehaving the magnification of N2 (in both images, the minimum squarerepresents 1 pixel, N1<N2).

[0277] Re-sampling is performed on the image having the magnification ofN2. The phase is matched such that the pixel at the upper left end ofthe image having the magnification of N2 coincides with a sampling pointof the image having the magnification of N1.

[0278] The re-sampling value is calculated by using an intersectionbetween the line of sight of the camera passing through-the point of theimage having the magnification of N1, and a two-dimensional curved planeconsisting of four points of the image having the magnification of N2surrounding said point.

[0279] In the above described embodiment, calculation of the parametersfor coordinate conversion is performed by using both the two-dimensionalcolor image and three-dimensional data. However, the coordinateconversion parameters can be calculated by using only thethree-dimensional data, without searching for the junction point fromthe two-dimensional color image. Though three-dimensional input has beendescribed in the present embodiment, this invention can be similarlyapplied to the two-dimensional image input.

[0280] A third embodiment of the present invention will be described inthe following.

[0281]FIG. 56 shows a basic structure of a three-dimensional shape inputapparatus. A light beam projected from a light source 201 has itsoptical path deflected by a first optical path deflecting apparatus 202such as a galvano scanner or a polygon scanner, extended in onedirection by means of a cylindrical lens 203 and thus the light beamwhich has been turned into a slit shaped light is directed to an object204. The slit shaped light is moved in a direction orthogonal to thelongitudinal direction of the slit shaped light for scanning, by meansof a first optical path deflecting apparatus 202. Further, the image towhich the slit shaped light is directed is photographed by asensingsystem 205 arranged spaced by a prescribed distance from thelight projecting optical system.

[0282] Actual measurement using the three-dimensional shape inputapparatus will be described referring to an example in which an imagehaving information of distance of 256 points in the longitudinaldirection of the slit shaped light and 324 points in the scanningdirection (hereinafter referred to as a distance image) is generated. Inthis case, the distance image sensor provided in the sensingsystem 205is constituted by a two-dimensional CCD area sensor having at least256×324 pixels.

[0283] The slit shaped light projected with a very narrow width is movedfor scanning by 1 pitch of the distance image sensor by means of thefirst optical path deflecting apparatus 202 while the distance imagesensor performs one image accumulation. The distance image sensorprovides the accumulated image information and performs next imageaccumulation. Based on the image information obtained by one imageaccumulation, the position of the centroid of the received lightintensity is calculated for each of 256 columns orthogonal to thelongitudinal direction of the slit shaped light. The calculated valueconstitute the distance information of 256 points in 1 pitch of thedistance image sensor. Since the image illuminated by the slit shapedlight is displaced in the direction of the scanning corresponding to theshape of the object, the obtained distance information represents theshape of the object at a position which is irradiated with the slitshaped light. When repeating this image accumulation for the number ofpitches of the distance image sensor, that is, 324 times, distance imagecorresponding to 256×324 points is generated.

[0284] Here, when the distance to the object is changed or when theangle of view for sensing by the distance image sensor (that is, focallength of the optical system) is changed, the region of the objectphotographed by the distance image sensor varies (details will bedescribed later). Therefore, when the scanning region of the slit shapedlights is kept constant in such a case, there would be a region notscanned, or regions outside the measurement region would be scanned. Forthis reason, the scanning region of the slit shaped light should beappropriately set in accordance with the distance to the object and thesensing angle of view.

[0285] The time for scanning by the slit shaped light for 1 pitch of thedistance image sensor must be sufficiently longer than the timenecessary for the distance image sensor to output the accumulated imageinformation. If the speed of scanning is too fast, the time ofaccumulation of image becomes too short, and the S/N ratio decreases,resulting in poor precision in calculation of the distance information.However, if the speed of scanning is too slow, the time for accumulationof the image becomes too long, possibly resulting in saturation of thesensor. This also leads to poor precision in calculating the distanceinformation. In view of the foregoing, the speed of scanning by the slitshaped light should be set at an appropriate constant value on the planeof the distance image sensor, that is, the imaging plane of thesensingsystem.

[0286]FIG. 57 shows a basic structure of an embodiment in which thescanning region and the scanning speed of the slit shaped light can bechanged. In the figure, the solid arrow denotes the row of information,while the dotted arrows show progress of the light beam and slit shapedlight.

[0287] The light beam projected from light source 201 has its pathdeflected by the first optical path deflecting apparatus 202. The firstoptical path deflecting apparatus 202 is driven at a prescribed timingand at a prescribed speed by a scan speed control apparatus 206.Further, a second optical path deflecting apparatus 207 on which angleof deflection can be changed is provided on the optical path. By meansof this apparatus, the light beam has its path further deflected toenter a cylindrical lens 203 in which the beam is extended to a slitshaped light, and finally directed to the object 204.

[0288] Meanwhile, the sensingsystem 205 includes an object distancedetecting apparatus 208 and an angle of view detecting apparatus 209,for detecting the distance to the object and the sensing angle of viewof the sensingsystem 5, respectively. A point of focus detectingapparatus used in an auto focus camera, for example, may be used as theobject distance detecting apparatus 208. An encoder provided at the lensdriving portion may be used, when the sensingsystem consist of a zoomlens unit, as the angle of view detecting apparatus 209. The objectdistance information output from object distance detecting apparatus 208and the sensing angle of view information output from the angle of viewdetecting apparatus 209 are taken in the calculating apparatus 210. Inthe calculating apparatus 210, the region of the field of view monitoredby the sensingsystem 205 at that point is estimated based on the objectdistance information and sensing angle of view information, and theapparatus determines the scan start angle and scan end angle forscanning the region thoroughly with the slit shaped light. A scanningscope control apparatus 211 adjusts the direction of projection of slitshaped light by driving the second optical path deflecting apparatus 207based on the scan start angle and scan end angle determined bycalculating apparatus 210, and adjusts light projection start time andlight projection end time by controlling the light source 201, thuscontrols the scanning scope with the slit shaped light. In calculatingapparatus 210, the speed of scanning by which the speed of movement ofthe slit-shaped image on the imaging plane of the sensingsystem comes tohave a prescribed value, is determined based on the determined scanningscope, and. based on this information, the scanning speed controlapparatus 206 drives the first optical path deflecting apparatus 202.

[0289] Namely, based on the object distance information and the sensingangle of view information, the speed of scanning with the slit shapedlight is controlled by the scanning speed control apparatus 206, and thescope of scanning with the slit shaped light is controlled by thescanning scope control apparatus 211, respectively.

[0290] By the above described structure, even when the object distanceor the sensing angle of view is changed, the sensing region of thesensingsystem 205 can be scanned thoroughly with the slit shaped light,and the speed of movement of the slit shaped light on the imaging planeis kept constant. Further, scanning of invalid region outside thesensing region can be avoided as much as possible. Therefore, whenmeasurement is to be continuously carried out, the time lag fromcompletion of one image input to the start of next image input can bemade very short.

[0291]FIG. 58 is an illustration of the third embodiment of the presentinvention. In this embodiment, a galvano scanner is used as the secondoptical path deflecting apparatus 207. The slit shaped light isprojected in a direction vertical to the sheet of paper. Now, assumethat the scanning region P1 with the slit shaped light and themonitoring region M1 are matched at a position of the object plane S1,and that the object plane moves to the position of S2. This time, theregion to be scanned is changed to the region P2, and the region to bephotographed is changed to M2, resulting in deviation between theregions. Accordingly, there will be a portion X which would not bescanned, in the region which is photographed. Accordingly, based on theresult of calculation by calculating apparatus 210 based on the objectdistance information detected by the object distance detecting apparatus208, the scanning scope control apparatus 211 changes the angle ofdeflection of the slit by driving the second optical path deflectingapparatus 207, and shifts the scan start angle and the scan end angle byθs and θe, respectively, by controlling the projection start time andprojection end time of the light source 201. This allows scanning of theregion P3, which corresponds to the sensing region M2.

[0292] Assume that the speed of scanning with the slit shaped light isconstant, then the speed of movement of the slit shaped light on theimaging plane of the sensingsystem becomes slower as the scanning regionbecomes larger (in this embodiment, the distance to the object becomeslonger), resulting in difference in measurement precision dependent onthe distance. Therefore, based on the newly determined scan start angleand the scan end angle, the calculating apparatus 210 calculates thespeed of scanning by which the speed of movement of the slit shapedlight on the imaging plane of the sensingsystem is kept at a prescribedvalue. Based on the result of calculation, the scan speed controlapparatus 206 controls the speed of driving of the first optical pathdeflecting apparatus 202. The first optical path deflecting apparatus202 is always driven under the condition in which the scanning angularregion is the largest, that is, in a deflection angle region whichcorresponds to the case where the distance to the object is the largest(in the measurable region).

[0293] Other than the reflective type apparatus such as a galvanoscanner, a prism of which angle of diffraction can be changed may beused as the second optical path deflecting apparatus 207 to obtain thesame effect. Further, the angular region of deflection by the firstoptical path deflecting apparatus 202 may be constant, and thereforewhen a rotary type scanner such as a polygon scanner is used, scanningat higher speed becomes possible.

[0294]FIG. 59 is an illustration of the fourth embodiment of the presentinvention. In this embodiment, the whole scanning system or part of thescanning system including a light source 201, scanning speed controlapparatus 202 and a cylindrical lens 203 is mounted on a movableapparatus 307, and the angle of mounting with respect to the wholeapparatus can be changed. The movable apparatus 307 serves as thescanning scope control apparatus.

[0295] Similar to the third embodiment, assume that the plane of theobject moves from the position S1 to the position S2. At this time,based on the object distance information detected by the object distancedetecting apparatus 208, the scanning scope control apparatus 211changes the angle of setting with respect to the entire apparatus bydriving the movable apparatus 30, whereby the angle of projection of theslit shaped light is changed. Further, the project start time and theproject end time of the light source 1 are controlled so that thescanning start angle and scanning end angle are shifted by θs and θe,respectively. Thus the region-scanned would be P3, which matches themonitoring region M2. The control for changing the scanning speed by thefirst optical path deflecting apparatus 202 is carried out in thesimilar manner as in the third embodiment.

[0296] Referring to the present embodiment, a rotary scanner such as apolygon scanner may be used as a first optical path deflecting apparatus202 as in the third embodiment, enabling scanning at higher speed.Further, since the scanning scope changing apparatus is not provided onthe optical path, the loss of the slit shaped light beam intensity canbe reduced. Meanwhile, similar effect can be obtained by fixing thescanning system and attaching the sensingsystem on a movable apparatusto change the angle of setting with respect to the whole apparatus suchthat the scanning region and the monitoring region match with eachother.

[0297]FIG. 60 is an illustration of the fifth embodiment of the presentinvention. In this embodiment, an apparatus in which the scan startangle, the scan end angle and the speed of scanning can be changed, forexample, a galvano scanner, is used as the first optical path deflectingapparatus 302.

[0298] In this embodiment also, assume that the plane of the objectmoves from the position S1 to the position S2, as in the thirdembodiment. At this time, based on the object distance detected byobject distance detecting apparatus 208, control apparatus 206/211controls the operation of the optical path deflecting apparatus 202/207so as to change the swing angle region from R1 to R2, and control theprojection start time and projection end time of the light source 1 toshift the scan start angle and scan end angle by θs and θe,respectively. As a result, the region P3 is scanned, which regionmatches the monitoring region M2. Control for changing the scanningspeed is performed in the similar manner as in the third and fourthembodiments.

[0299] The fifth embodiment can be-regarded as implementation of thescanning speed control apparatus 206 and scanning scope controlapparatus 211 of the third embodiment by one apparatus that is, controlapparatus 206/211 and implementation of the first optical pathdeflecting apparatus 202 and the second optical path deflectingapparatus 207 by one apparatus, that is, optical path deflectingapparatus 202/207. Therefore, the structure of the apparatus is madesimple.

[0300]FIG. 61 shows control when the angle of view for sensing of thesensingsystem 205 is changed in the fifth embodiment described above.Now, assume that from the state in which the scanning region P1 matchesthe monitoring region Ml with the angle of view of the sensingsystem 205being Φ1, the angle of view of the sensingsystem 205 is changed to Φ2,that is, to wide angle side. At this time, the monitoring region wouldbe M2, so that there is difference between the scanning region and themonitoring region, and hence portions X and X′ which are not scannedwould exist in the monitoring region, hindering successful measurement.Therefore, based on the view angle information-detected by the viewangle detecting apparatus 209, control apparatus 206/211 controls theoperation of the optical path reflecting apparatus 202/207 so as tochange the rotation angle from R1 to R2, and the projection start timeand projection end time of the light source 201 are controlled so as toshift the-scan start angle and scan end angle by θs and θe,respectively. Consequently, the region to be scanned would be P2, whichmatches the monitoring region M2. It goes without saying that the speedof scanning is changed under the control of optical path reflectingapparatus 202/207.

[0301]FIG. 62 is an illustration taking into consideration the depth Dof the object in the fifth embodiment. Though it depends on theconditions of setting the object distance detecting apparatus 208, thedistance detected by the object distance detecting apparatus 208 is inmost cases, a position near the center of field of view, for example,the point C. However, when the plane of the object S1 is positioned atthis point C, the scanning region would be P1 with respect to themonitoring region M1, and therefore the depths of the object cannot betaken into account, resulting in a portion X which is not scanned.Therefore, to the object distance detected by the object distancedetecting apparatus 208, an offset Ad taking into account the depth isadded, and the result 1 s regarded as the object distance. By thisoperation, referring to FIG. 62, the plane of the object is assumed tobe at the position S2. The scanning region for the position S2 is P2,which can cover the depth of the object. The amount of offset Ad can bedetermined in the following manner, for example. Now, in measurement,let us assume that a constant depth corresponding to −K1 pixel −K2pixel, in the direction of scanning, that is, depth corresponding to thewidth of K1+K2 pixels should be ensured for an arbitrary pixel on theimage pickup device of the sensingsystem. At this time, in order to setthe object distance dl detected by the object distance detectingapparatus 208 coincide with the limit S1 of the depth closest to thesensingsystem, a virtual object plane S2 should be placed at a distanced2 provided geometrically by the following equation:

d2=α/tan(arctan(α/d1)−K1·Δθ)

[0302] where the scanning angle per 1 pixel in the slit scanningdirection of the image pickup device of the sensingsystem 205 isrepresented by Δθ, and the base length, which is a space in a directionvertical to the optical axis of the sensingsystem, between the mainpoint of the light emitting scanning system and the main point of thesensingsystem is represented by α. Therefore, the amount of offset isobtained by

Δd=d2−d1=α/tan(arctan(α/d1)−K1·Δθ)−d1

[0303] At this time, the limit d3 of the depth which is farthest fromthe sensingsystem is given by the following equation:

d3=α/tan(arctan(α/d1)−K·Δθ−K2·θ).

[0304] Example of a method for determining scan start angle, scan endangle and scanning speed will be described with reference to the fifthembodiment. Referring to FIG. 64, α represents the base length which isa space in the Y direction between the main point of the light emittingscanning system and the main point of the sensingsystem; doff representsoffset in the Z direction which is the space in the Z direction; drepresents the object plane distance; i represents size (image size) ofthe distance image sensor used in the sensingsystem; δ representsover-scan amount for scanning slightly wider region than the lightreceiving field of view, in order to ensure the depth forthree-dimensional detection at end portion corresponding to start andend of the scanning, similar to the central portions; np represents thenumber of effective pixels of the image sensor in the Y direction, and frepresents focal length of the sensingsystem. At this time, the startangle th1, scan end angle th2 and scan angular speed ω are given by thefollowing equations:

th1(°)=arctan[{d(i/2+δ)/f+α}/(d+doff)]×180/π

th2(°)=arctan[{−d(i/2+δ)/f+α}/(d+doff)]×180/π

ω=k·(th1−th2)/np (k is a constant).

[0305] The calculated values th1 and th2 are shown in FIG. 65, in whichf is used as a parameter and the abscissa represents the object planedistance. Similarly, the calculated value ω is shown in FIG. 66. In thisembodiment, the image size is assumed to be ½ inch, the constant k=1 andthe base length α=250 mm. Because of this base length, there is aparallax between the scanning system and the sensingsystem, andtherefore the start angle and end angle vary widely dependent on theobject plane distance. The ordinate represents the angle formed by theoptical axis of the sensingsystem and the projected slit.

[0306] In the above described embodiments, the scanning scope of theslit-shaped light beam (scanning direction and scan start angle and scanend angle) is changed in accordance with the distance to the object orin accordance with the sensing angle of view. However, it is possible toscan a sufficiently large area with the slit shaped light so as to coverentire scanning region (that is, entire field of view of thesensingsystem) which may fluctuate due to the change in the distance tothe object or the change in the sensing angle of view. In that case,unnecessary region outside the measurement region may be scanned.However, mechanism and control necessary for scanning only themeasurement region becomes unnecessary, and therefore the apparatus canbe simplified.

[0307]FIG. 67 shows a specific structure of an apparatus in accordancewith the sixth embodiment in which only the change in the scanning speedof the slit shaped light is possible (the scanning scope is alwaysconstant). As in FIG. 57, the solid arrow represents the flow ofinformation, while a dotted arrow represents progress of the light beamand the slit shaped light. What is different from FIG. 57 is that thereis not the scanning scope control apparatus 211 and the second opticalpath deflecting apparatus 207 provided for changing the scanning scope.

[0308]FIGS. 68 and 69 are illustrations of the seventh embodiment of thepresent invention. This embodiment is based on the fifth embodimentabove, and differs in that the optical deflecting apparatus 202 is notdriven for changing the scanning scope. FIG. 68 shows an example inwhich the distance to the object is changed, and FIG. 69 shows thechange in the sensing angle of view, which correspond to FIGS. 60 and 61of the fifth embodiment, respectively. For controlling the scanningspeed, equations given above can be directly used.

[0309] Table 1 below shows apparatuses for actually controlling thescanning scope and the scanning speed in the third to fifth and seventhembodiments. TABLE 1 Control of Scanning Scope Control of EmbodimentFig. Direction Start-End Scanning Speed 3rd Embodiment 207 201 202 4thEmbodiment 307 201 202 5th Embodiment 302 201 302 7th Embodiment NONENONE 303

[0310] Next, the problem that the number of pixels receiving the lighton the light receiving element changes when the sensing angle of viewchanges while the width of slit shaped light is kept constant, will bediscussed.

[0311] In order to detect the position of the slit with high accuracy,it is preferable that the width of the slit viewed by the sensingsystemand the distribution of light intensity are always kept constant. It ispossible to calculate the centroid of the slit shaped light in thewidthwise direction when the width of the slit shape light changes.However, since the width of the slit varies dependent on the angle ofview, the precision in calculating the centroid, that is the precisionin measurement, would also be dependent on the angle of view, which isnot preferable. Assume that the slit shaped light has approximatelyGaussian distribution, for example. Then, the precision in calculatingthe centroid is poor when the slit shaped light is narrow and the numberof pixels receiving the light beam is too small (FIG. 70), and theprecision in-calculating the centroid is also poor when the slit shapedlight beam is too wide and the number of pixels receiving the light istoo many (FIG. 71). Therefore, the width of the slit shaped light shouldpreferably have a constant width of several pixels on the lightreceiving device, regardless of the angle of field of the lightreceiving lens.

[0312] For example, when the sensing region changes from region A toregion B of FIG. 72 by the zooming operation of the light receiving lenswhile the width of the slit shaped light does not change in relation tothe change of the angle of view, the light receiving region on the lightreceiving plane such as the area sensor would be changed from the stateof FIG. 73(a) to FIG. 73(b) (Quantatively, it would be changed by thesame amount as the zooming ratio). Consequently, the number of lightreceiving pixels in the width direction changes, resulting in variationof precision in measurement dependent on the angle of view. If thezooming ratio is large, there would be an angle of view at whichmeasurement becomes impossible.

[0313] There is also a problem generated as the sensing angle of viewchanges in the longitudinal direction of the slit shaped light. Forexample, when the sensing region is changed from region A to region B inFIG. 70, the slit shaped light adjusted to illuminate the region Aappropriately would illuminate the region B as well as unnecessaryregion surrounding the region B, which is wasteful.

[0314]FIG. 74 shows an eighth embodiment of the present invention.Referring to FIG. 74, a collimator lens 22 is provided below a lightsource (hereinafter referred to as LD) 21 such as a semiconductor laser,for receiving the luminous flux from the LD and emitting the luminousflux with a prescribed angular extension near parallel flux. A mask 23regulates the luminous flux incident on the collimator lens. The maskintercepts light beam out of the Gaussian distribution, from the lightbeam emitted from the laser light source. Consequently, a light beam ofwhich light intensity has Gaussian distribution is obtained, and hencethe received light also comes to have approximately Gaussiandistribution.

[0315] Lenses 324 and 325 change the length and the width of theprojected slit shaped light, and a cylindrical lens (A) 324 hascurvature only in one direction. A cylindrical lens (B) 325 hascurvature in a direction orthogonal to the direction of curvature ofcylindrical lens (A) 324. By using two or more cylindrical lenses, theslit shaped light can be readily generated of which width and length canbe freely controlled. More specifically, the collimator lensmonotonously changes the diameter of the emitted luminous flux in thedirection of the optical axis, so that when the position of thecylindrical lens is changed in the direction of the optical axis, theincident height to the cylindrical lens changes, and hence the shape ofthe slit shaped light can be changed. Therefore, the shape of the slitshaped light, that is, width and length can be arbitrarily controlled bya simple structure.

[0316] For example, when the position of cylindrical lens A changes fromposition a to position b of FIG. 75 by the distance D1, the incidentheight to the cylindrical lens A and the incident angle to the curvedsurface Cl of the light beam L1 (outermost light beam of the emittedluminous flux) emitted at an angle δ from the collimator lens changed,and hence the emission angle of cylindrical lens A changes from an angleθa1 to θa2 with respect to the optical axis. The same applies to thecylindrical lens B. Therefore, by driving the cylindrical lenses (A) and(B) in the direction of the optical axis, the shape of the slit shapelight on the object can be changed to an arbitrary shaped.

[0317] The curvature of each cylindrical lens is determined based on theamount of driving of the cylindrical lens and the ratio of change of theshape of the slit shaped light as it is driven. At this time, thedistance between the collimator lens and the cylindrical lens and theemission angle of the luminous flux from the collimator lens maypreferably be referred to as parameters, so as to facilitate control ofdriving two cylindrical lenses. For example, when the proportion ofdriving gears of two cylindrical lenses are selected to be the same, thetwo lenses can be driven by one driving source, enabling reduction insize of the apparatus and reduction in power consumption. The twocylindrical lenses are each held in a holder (not shown), and the holderis connected to the driving source through driving means such as aball-like screw. A rack and a pinion or a cam may be used as the drivingmeans.

[0318] For example, optical scanning means 326 such as a galvano mirroris arranged close to the object in the optical path. By thisarrangement, highly linear slit shaped light can be projected,regardless of the direction angle of the projected slit shaped light. Bycontrast, if the cylindrical lens is arranged nearer to the object thanthe optical scanning means and the cylindrical lens has general shape,end portions of the slit shaped light may possible be deformed,dependent on the angle of projection. In order to avoid such a problem,the shape of the cylindrical lens must be arcuate with the start pointof scanning being the center, resulting in larger lens and largerapparatus as a whole. Therefore, the arrangement of the optical systemin accordance with this embodiment realizes reduction in size of thecylindrical lens and of the three-dimensional measuring apparatus. Thelight optical scanning means may be a rotary polygon mirror.

[0319] In the present invention, prior to measurement of thethree-dimensional shape, the image obtained at the light receivingdevice is displayed on a monitor and framing of the image is performed.During framing, the operator monitors the image and changes thedirection of the measuring apparatus, and position and focal distance ofthe light receiving lens. When the focal length (that is, sensing angleof view) of the light receiving lens is changed by zooming, a signal istransmitted from an angle of view detecting means detecting the changein the angle of view based on the position of the light receiving lensto the driving amount control portion. Based on the transmitted signal,the driving amount control portion calculates the amount of drivingcylindrical lenses (A) and (B), provides a driving signal, and drivesthe cylindrical lenses.

[0320] By this method, the shape of the beam can be optimized withouttroublesome operation by the user. For example, when the magnificationchanges from β1 (region A of FIG. 76) to β2 (region B of FIG. 76) bychanging the angle of view of the light receiving lens, the cylindricallenses A and B are driven such that the width W and length L of the slitshaped light attain W×(β1/β2) and L×(β1/β2), that is, the values beforezooming are multiplied by β1/β2. As a result, the width and length ofthe slit on the light receiving device are always kept constantregardless of the zooming of the light receiving lens, as shown in FIG.77. Therefore, three-dimensional shape can be measured while there ishardly a variation in precision caused by zooming.

[0321] When the light receiving lens has high magnification rate, thechange in size of the slit shaped light is also large. Therefore, whenLD having a prescribed constant output is used, the change in the amountof exposure at the light receiving device is also large. Therefore,exposure amount adjusting means for adjusting the amount of exposurebecomes necessary. In this embodiment (FIG. 78), the amount of exposureis adjusted by an LD output control portion 1. For example, when themagnification of the sensingsystem changes from β1 to β2 by β12 (=β2/β1)and the area of the slit shaped light changes by the square of (1/β12),the amount of light on the light receiving device become square times(1/β12). Therefore, in the present embodiment, when magnification β12 iscalculated from the output of the angle of view detecting portion 352,the LD output is controlled by the LD output control portion (1) 354 sothat the LD output attains square times (β12), as the necessary amountof exposure is square times (β12) before the change of the angle ofview. By this method, the amount of exposure can be adjusted without anyadditional mechanical structure, and therefore it is not expensive.Further, even when the light receiving lens for the slit shaped light isalso used as a light receiving lens for framing, the amount of exposurecan be adjusted independent from the amount of exposure at the lightreceiving device for framing, and therefore measurement can be done withoptimal amount of exposure.

[0322] As a modification of the exposure amount adjusting means, a gaincontrol portion 356 for calculating and controlling the gain of thelight receiving device which is necessary for obtaining appropriateamount of exposure from the output of the angle of view detectingportion may be provided at the light receiving device. The calculationof the gain is as follows.

[0323] For example, when the magnification changes from β1 to β2 by β12(=β2/β1) and the area of the slit shaped light changes square times(1/β12), then light intensity on the light receiving device is squaretimes (1/β12). Therefore, the gain is controlled by the gain controlportion 356 so that the gain becomes square times (β12) of the valuebefore the change of the angle of view. By this method, the gain can beadjusted without any additional mechanical structure, and therefore itis inexpensive. Further, even when the light receiving lens for the slitshaped light is also used as the light receiving lens for framing,adjustment can be performed independent from the amount of exposure atthe light receiving device for framing, and therefore measurement can bedone with optimal amount of exposure.

[0324] As another modification of the exposure amount adjusting means, adiaphragm may be provided on the entrance side of the light receivingelement, and a diaphragm control portion for calculating and controllingthe amount of stepping down of the diaphragm necessary for obtaining theappropriate amount of exposure from the output of the angle of viewdetecting portion 352 may be provided at the light receiving apparatus.For example, when the magnification changes from β1 to β2 by β12(=β2/β1) and the area of the slit shaped light changes square times(1/β12), the amount of light on the light receiving device becomessquare times (1/β12). Therefore, the calculated amount of stepping downof the diaphragm is the value before the change of the angle of viewtimes (β12), in terms of the area of opening.

[0325] As a further modification of the exposure amount adjusting means,an amount of exposure detecting portion 358 for determining whether ornot the amount of exposure at the light receiving device is lower thanthe threshold values set at a threshold value setting portion may beprovided, and when it is determined that the amount of exposure is lowerthan the threshold value, the output of LD may be controlled so that theLD output exceeds the threshold value. By this method, the amount ofexposure can be adjusted without any additional mechanical structure,and therefore it is not expensive. Even when the light receiving lensfor the slit shaped light is also used as the light receiving lens forframing, the adjustment can be carried out independent from the amountof exposure for the light receiving device for framing, and thereforemeasurement can be done with optimal amount of exposure.

[0326] One of the above described several means for adjusting amount ofexposure may be used by itself, or some of these means may be used incombination. FIG. 82 is a flow chart showing an operation when LD outputcontrol portion 1, the gain control portion and the diaphragm controlportion are provided as means for adjusting the amount of exposure.

[0327] Though two cylindrical lenses are used in the eighth embodiment,a structure employing an anamorphic lens is also possible. In this case,the degree of freedom is reduced compared with the example using two ormore cylindrical lenses. However, by arranging a beam expander havingcylindrical axis in the same direction as either of the cylindricallenses between the collimator lens and the anamorphic lens, a desiredprojection angle is obtained using h and γ of FIG. 75 as parameters. Bythis method, the number of cylindrical lenses can be reduced to 1, andonly one driving portion and only one driving source are necessary.Therefore, the apparatus can be made compact and the cost ofmanufacturing can be reduced.

[0328]FIG. 83 shows a ninth embodiment of the present invention.Compared to the eighth embodiment, in th ninth embodiment, there arethree light emitting portions 31, three collimator lenses 32, threemasks 33 and three cylindrical lenses (A) 34. The light beam emittedfrom three light emitting portions 431 a to 431 c are adapted such thatthe light beam passed through the cylindrical lens (b) 435 and thenprojected as one slit. Therefore, only one cylindrical lens 435 issufficient, and the cost can be reduced and adjustment is simple. Sincethe beams are turned to one slit shaped light after passing through thecylindrical lens 435, only one optical scanning means 436 is sufficient,and therefore the number of parts can be reduced, the size of theapparatus can be reduced and the manufacturing cost can also be reduced.Referring to FIG. 34, the relation between the extension angle i in thelongitudinal direction of the slit after the passage through cylindricallens (B) and the angle j provided by main axis of adjacent slits ismaintained such that part of each slit are overlapped on the plane ofprojection irradiated with the slit shaped light.

[0329] Assuming that the beam intensity has Gaussian distribution, lightintensity with outer portion having higher intensities such as shown inFIG. 85 can be obtained by adjusting the angle k formed by outer twobeams is close to the angle of view of the field of view and byadjusting the ratio of outputs of the outer beams and the central beam.By this intensity distribution, reduction of the amount of light at theperiphery derived from cosine fourth law and shading after passagethrough the light receiving lens can be compensated for. As a result,three-dimensional shape can be measured with high precision even at theedges of the sensing region.

[0330] This embodiment includes, as shown in FIG. 86, a threshold angleof view setting portion for setting the threshold angle of view at whichthe field of view cannot be covered by projection by one slit, an angleof view comparing portion for comparing the set threshold angle of viewand the value of the angle of view from the angle of view detectingportion, and an LD on/off control portion for controlling on/off ofthree LDs. For example, when it is found that one LD is not enough tocover the field of view as a result of the comparison, three LDs are allturned on by the LD on/off control portion, so that light is projectedto the entire field of view for measurement (FIG. 87). By thisembodiment, the driving portion can be eliminated, and therefore thepower consumption can be reduced, manufacturing cost can be reduced asthe number of part is reduced, and the size of the apparatus can be madesmaller. FIG. 88 is a flow chart of operation of this embodiment.

[0331] Although the present invention has been described and illustratedin detail, it is clearly understood that the same is by way ofillustration and example only and is not to be taken by way oflimitation, the spirit and scope of the present invention being limitedonly by the terms of the appended claims.

What is claimed is:
 1. A three-dimensional measuring apparatuscomprising: an emitting portion for emitting reference light of apredetermined intensity; a sensor for sensing light including thereference light reflected from an object; an optical system which has afocal length and through which the reflected light reaches the sensor;and a controller for controlling the intensity of the reference light inaccordance with the focal length of the optical system.
 2. Athree-dimensional measuring apparatus comprising: an emitting portionfor emitting reference light of a predetermined intensity; a sensor forsensing light including the reference light reflected from a object; azoom lens through which the reflected light reaches the sensor; and acontroller for controlling the intensity of the reference light inaccordance with a magnification ratio defined by the zoom lens.
 3. Athree-dimensional measuring apparatus comprising: an emitting portionfor emitting reference light of a predetermined intensity; a capturingportion for capturing light including the reference light reflected froman object; an obtaining portion for obtaining a distance datacorresponding to a distance from the apparatus to the object; and acontroller for controlling the intensity of the reference light inaccordance with the obtained distance data.
 4. A three-dimensionalmeasuring apparatus comprising: an emitting portion for emittingreference light of a predetermined intensity; a capturing portion forcapturing light including the reference light reflected from an object,the capturing portion having a predetermined optical condition; and acontroller for controlling the intensity of the reference light inaccordance with the predetermined optical condition.
 5. Thethree-dimensional measuring apparatus as claimed in claim 4 , whereinthe capturing portion includes a sensor for sensing the light and anoptical system through which the light reaches the sensor, thepredetermined optical condition including a condition of the opticalsystem.
 6. The three-dimensional measuring apparatus as claimed in claim5 , wherein the optical system has a variable focal length, and whereinthe condition of the optical system includes a focal length defined bythe optical system.
 7. The three-dimensional measuring apparatus asclaimed in claim 5 , wherein the optical system has a zooming function,and wherein the predetermined optical condition includes a magnificationratio defined by the zooming function of the optical system.
 8. Thethree-dimensional measuring apparatus as claimed in claim 4 , whereinthe emitting portion has a plurality of light emitting elements foremitting the reference light, wherein the controller determines a numberof the plurality of light emitting elements being in an on-state foremitting the reference light to control the intensity of the referencelight.
 9. The three-dimensional measuring apparatus as claimed in claim4 , wherein the emitting portion includes a laser light source foremitting the reference light and a power supply for supplying power tothe laser light source, and wherein the controller controls the powersupplied to the 11 laser light source from the power supply to controlthe intensity of the reference light.
 10. The three-dimensionalmeasuring apparatus as claimed in claim 4 , further comprising a filter,wherein the emitting portion includes a laser light source for emittingthe reference light, wherein the controller controls the filter so thatthe reference light passes through the filter or does not pass throughthe filter.
 11. The three-dimensional measuring apparatus as claimed inclaim 4 , wherein the reference light is a slit shaped light, whereinthe emitting portion includes a scanning mechanism for deflecting theslit shaped light in a direction perpendicular to a longitudinaldirection of the slit shaped light, and wherein the capturing portionsequentially captures the reflected light while scanning.
 12. Athree-dimensional measuring apparatus comprising: an emitting portionfor emitting reference light of a predetermined intensity; a capturingportion for capturing light including the reference light reflected froman object, the capturing portion having a predetermined opticalcondition and outputting a signal corresponding to the captured light;an amplifier for amplifying a signal to be processed forthree-dimensional measurement; and a controller for controlling theintensity of the emission in accordance with the predetermined opticalcondition of the capturing portion.
 13. The three-dimensional measuringapparatus as claimed in claim 12 , wherein the capturing portionincludes a sensor for receiving the reference light reflected from theobject and an optical system through which the reference light reflectedfrom the object reaches the sensor.
 14. The three-dimensional measuringapparatus as claimed in claim 13 , wherein the optical system has avariable focal length, and wherein the predetermined optical conditionincludes a focal length defined by the optical system.
 15. Thethree-dimensional measuring apparatus as claimed in claim 12 , whereinthe optical system has a zooming function, and wherein the predeterminedoptical condition includes a magnification ratio defined by the zoomingfunction.
 16. The three-dimensional measuring apparatus as claimed inclaim 12 , wherein the reference light is a slit shaped light, whereinthe emitting portion has a scanning mechanism for deflecting the slitshaped light in a direction perpendicular to a longitudinal direction ofthe slit shaped light, and wherein the capturing portion sequentiallycaptures light during scanning.
 17. A three-dimensional measuringapparatus comprising: an emitting portion for emitting reference lightof a predetermined intensity; a scanning mechanism for deflecting thereference light to scan an object; a sensor for sensing light includingreference light reflected from the object; an operation portion foraccepting an instruction relating to modification of the intensity ofthe reference light; and a controller for controlling the intensity ofthe reference light according to the accepted instructions.